Building

Started building at 2022/09/10 13:58:19
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 10.09.2022-13.58
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at de1130b, changes since last good build: none
Switching plasma to unstable
plasma is at a338d0e, changes since last good build: 
fatal: Invalid revision range 2455f58ea891433394e537dcd7d420fc1e15b74e..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable 53f87c78] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 6d54cb83] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 1bb72ddd] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable ae62817d] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable cab097e0] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 9d92bba4] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable 0145ab28] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable 5bc4298d] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable adbe09b9] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 422168d3] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable e44c4871] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 6ec4c602] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179227 at refs/changes/27/179227/10
Total 8 (delta 3), reused 4 (delta 3)
[unstable f6e895b4] MB-51825: Remove vbuckets arg from indexer command line and its usage
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Aug 25 18:17:32 2022 +0530
 2 files changed, 9 deletions(-)
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 862be331] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 5f2d012] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable eff6826] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 1387aca] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 5fd56f7] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 7f94e72] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable dc0ffb0] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable f76553f] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 21198ff] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Pulling in uncommitted change 179769 at refs/changes/69/179769/2
Total 5 (delta 4), reused 5 (delta 4)
[unstable b25e4a2] MB-52096: Use memory stats to estimate mutation quota if page stats not available
 Author: jliang00 
 Date: Thu Sep 8 16:56:16 2022 -0700
 3 files changed, 86 insertions(+), 27 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/10 14:52:33
Testing mode: sanity,unit,functional,integration
Using storage type: fdb
Setting ulimit to 200000

Simple Test

Sep 10 14:57:58 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 15:01:51 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 15:02:36 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 10 15:04:01 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 10 15:04:51 b" 'stop_on_failure': 'True'}"
Sep 10 15:04:51 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 15:04:51 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_1'}"
Sep 10 15:04:51 b'-->result: '
Sep 10 15:04:51 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 10 15:04:51 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 15:04:51 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_2'}"
Sep 10 15:04:51 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 15:04:51 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 10 15:04:51 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_3'}"
Sep 10 15:04:51 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 15:04:51 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 15:04:51 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 10 15:04:51 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_4'}"
Sep 10 15:04:51 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 15:04:51 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 15:04:51 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 15:14:57 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 10 15:15:43 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 10 15:19:28 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 15:23:24 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 15:26:06 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 15:26:06 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_5'}"
Sep 10 15:26:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 15:26:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 10 15:26:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 10 15:26:06 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_6'}"
Sep 10 15:26:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 15:26:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 15:26:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 10 15:26:06 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_7'}"
Sep 10 15:26:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 15:26:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 15:26:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 10 15:26:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 10 15:26:06 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_14-52-55/test_8'}"
Sep 10 15:26:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 15:26:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 15:26:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 15:26:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 10 15:26:06 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 10 15:26:07 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 15:26:07 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 15:26:07 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 10 15:26:07 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 10 15:26:07 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 10 15:26:07 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 10 15:26:07 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 10 15:26:07 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=forestdb
Printing gsi_type=forestdb
gsi_type=forestdb
In here
-p makefile=True,gsi_type=forestdb
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'forestdb',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-10 15:28:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:28:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:28:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:28:55 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 15:28:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 15:29:04 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 15:29:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 15:29:23 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:29:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 15:29:24 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 15:29:24 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 15:29:24 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 15:29:24 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:24 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-10 15:29:25 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 2081792000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:29:25 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:29:25 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:29:25 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 13534281728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:29:25 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 15:29:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:29:26 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:29:26 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1790996480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:29:26 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 15:29:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:29:27 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:29:27 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1681158144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:29:27 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 15:29:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:29:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:29:28 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:29:28 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 15:29:28 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 15:29:28 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-10 15:29:29 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:29:34 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 15:29:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 15:29:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-10 15:29:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 15:29:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 15:29:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 15:29:36 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 0, 'mem_free': 13423656960, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.62533187866211
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:29:39 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:29:40 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:29:40 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:29:40 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:29:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-10 15:29:40 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 15:29:40 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 15:29:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 15:29:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:29:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:29:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:29:46 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 15:29:46 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 15:29:48 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 15:29:48 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:29:48 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 15:29:48 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 15:30:18 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 15:30:18 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 15:30:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 15:30:18 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 131.803105ms
2022-09-10 15:30:18 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '350d080d-297e-4c29-8edb-7ab2659601f3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '131.803105ms', 'executionTime': '131.664308ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 15:30:18 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 15:30:18 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 15:30:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 767.241441ms
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.45004ms
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.398752ms
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 53.204472ms
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.847751670211699, 'mem_free': 13392461824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:25 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 15:30:25 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 15:30:26 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 15:30:26 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 91.853s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.04810385249825, 'mem_free': 13385416704, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 15:30:32 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 15:30:32 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 15:30:33 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13363978240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 15:30:34 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:30:34 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13353369600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:30:35 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13368631296, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:30:35 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13366611968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:36 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:36 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:30:36 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:30:36 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:30:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 15:30:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 15:30:36 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 15:30:36 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 15:30:36 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 15:30:41 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 15:30:41 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-10 15:30:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 15:30:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 15:30:42 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 15:30:42 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 15:30:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 15:30:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-10 15:30:42 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 15:30:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 15:30:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 15:30:44 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.15322145089313, 'mem_free': 13354835968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.259809970855713
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 15:30:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 15:30:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 15:30:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:30:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:30:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:30:54 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 15:30:54 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 15:30:56 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 15:30:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:30:56 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 15:30:56 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 15:31:26 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 15:31:26 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 15:31:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 15:31:26 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.024751ms
2022-09-10 15:31:26 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '28d7eed4-95e9-4e48-995b-3c73f06e7dd8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.024751ms', 'executionTime': '67.915222ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 15:31:26 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 15:31:26 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 15:31:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 703.415296ms
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.158443ms
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 15:31:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 15:31:28 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:31:28 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeea942e83e53954d31b7d1f7ad46a955cajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-10 15:31:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeea942e83e53954d31b7d1f7ad46a955cajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-10 15:31:28 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 62.698237ms
2022-09-10 15:31:28 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeea942e83e53954d31b7d1f7ad46a955cajob_title) USING GSI
2022-09-10 15:31:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeea942e83e53954d31b7d1f7ad46a955cajob_title) USING GSI
2022-09-10 15:31:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeea942e83e53954d31b7d1f7ad46a955cajob_title%29+USING+GSI
2022-09-10 15:31:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.126231ms
2022-09-10 15:31:31 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea942e83e53954d31b7d1f7ad46a955cajob_title'
2022-09-10 15:31:31 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea942e83e53954d31b7d1f7ad46a955cajob_title%27
2022-09-10 15:31:31 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.804657ms
2022-09-10 15:31:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea942e83e53954d31b7d1f7ad46a955cajob_title'
2022-09-10 15:31:32 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea942e83e53954d31b7d1f7ad46a955cajob_title%27
2022-09-10 15:31:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.92831ms
2022-09-10 15:31:33 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 15:31:33 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 15:31:33 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-10 15:31:33 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.185187ms
2022-09-10 15:31:33 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '9c4bdb8e-751f-4a49-99e2-26f08dd8edd1', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea942e83e53954d31b7d1f7ad46a955cajob_title', 'index_id': '8afaaf34d15c834f', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.185187ms', 'executionTime': '2.127519ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-10 15:31:33 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 15:31:33 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 15:31:33 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 15:31:33 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-10 15:31:33 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-10 15:31:33 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-10 15:31:33 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-10 15:31:33 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-10 15:31:33 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-10 15:31:34 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 15:31:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 15:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-10 15:31:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 141.432736ms
2022-09-10 15:31:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-10 15:31:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-10 15:31:35 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 15:31:35 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea942e83e53954d31b7d1f7ad46a955cajob_title'
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea942e83e53954d31b7d1f7ad46a955cajob_title%27
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.552272ms
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeea942e83e53954d31b7d1f7ad46a955cajob_title ON default USING GSI
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeea942e83e53954d31b7d1f7ad46a955cajob_title+ON+default+USING+GSI
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 37.382918ms
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea942e83e53954d31b7d1f7ad46a955cajob_title'
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea942e83e53954d31b7d1f7ad46a955cajob_title%27
2022-09-10 15:31:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.585638ms
2022-09-10 15:31:36 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'aeae30d5-2968-4101-9098-542819b17765', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.585638ms', 'executionTime': '6.481489ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:31:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.260844ms
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 48.187891ms
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.820912410764689, 'mem_free': 13275779072, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:31:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:31:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:31:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:31:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:31:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:31:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 15:31:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 15:31:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 15:31:42 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 15:31:42 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 15:31:44 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 15:31:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.668s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_2'}
[2022-09-10 15:31:44,723] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:44,723] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:45,011] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:45,043] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:31:45,127] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:31:45,128] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-10 15:31:45,128] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:31:45,753] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:31:45,780] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13248618496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:31:45,807] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:31:45,807] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:31:45,808] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:31:45,846] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:31:45,881] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:31:45,881] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:31:45,910] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:31:45,911] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:31:45,911] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:31:45,911] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:31:45,960] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:31:45,964] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:45,964] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:46,250] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:46,251] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:31:46,322] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:31:46,323] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:31:46,353] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:31:46,354] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:31:46,380] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:31:46,467] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:31:46,494] - [task:165] INFO -  {'uptime': '164', 'memoryTotal': 15466930176, 'memoryFree': 13281701888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:31:46,521] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:31:46,549] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:31:46,549] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:31:46,601] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:31:46,604] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:46,604] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:46,878] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:46,879] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:31:46,951] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:31:46,952] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:31:46,984] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:31:46,985] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:31:47,012] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:31:47,095] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:31:47,126] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13263360000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:31:47,154] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:31:47,182] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:31:47,183] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:31:47,233] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:31:47,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:47,237] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:47,507] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:47,508] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:31:47,580] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:31:47,581] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:31:47,610] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:31:47,611] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:31:47,636] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:31:47,723] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:31:47,750] - [task:165] INFO -  {'uptime': '160', 'memoryTotal': 15466930176, 'memoryFree': 13280362496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:31:47,775] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:31:47,804] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:31:47,804] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:31:47,855] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:31:47,858] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:47,858] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:48,118] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:48,119] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:31:48,187] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:31:48,188] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:31:48,218] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:31:48,218] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:31:48,244] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:31:48,299] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:31:48,796] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:31:53,799] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:31:53,885] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:31:53,889] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:53,889] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:54,184] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:54,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:31:54,253] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:31:54,255] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:31:54,255] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:31:55,417] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:31:55,474] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 15:31:55,474] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:31:56,472] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:31:56,768] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:31:57,051] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:31:57,053] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:31:57,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:57,106] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:57,377] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:57,382] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:57,382] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:57,646] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:57,651] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:57,651] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:57,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:31:57,926] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:31:57,927] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:31:58,321] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:02,010] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:32:02,011] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.44170659742169, 'mem_free': 13263142912, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:32:02,011] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:32:02,012] - [basetestcase:467] INFO - Time to execute basesetup : 17.29080891609192
[2022-09-10 15:32:02,062] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:32:02,063] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:32:02,115] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:32:02,115] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:32:02,172] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:32:02,172] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:32:02,227] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:32:02,227] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:32:02,278] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:32:02,342] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:32:02,343] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:32:02,343] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:32:07,352] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:32:07,356] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:32:07,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:32:07,630] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:08,554] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:32:08,717] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:32:10,370] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:32:10,391] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 15:32:11,038] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:32:11,038] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:32:11,038] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:32:41,062] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:32:41,092] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:32:41,118] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:32:41,186] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.039487ms
[2022-09-10 15:32:41,187] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'cf699e48-7889-4734-9949-681b93398b2e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.039487ms', 'executionTime': '66.982299ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:32:41,187] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:32:41,213] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:32:41,240] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:32:41,906] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 664.727459ms
[2022-09-10 15:32:41,907] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:32:41,968] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:32:42,007] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:32:42,016] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.302956ms
[2022-09-10 15:32:42,214] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:32:42,251] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:32:42,275] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:32:42,277] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:32:42,290] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:32:42,351] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:32:43,168] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee8e0fd50a402c49edbfc8efbdc0402ec1job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:32:43,193] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee8e0fd50a402c49edbfc8efbdc0402ec1job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:32:43,246] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.56461ms
[2022-09-10 15:32:43,246] - [base_gsi:282] INFO - BUILD INDEX on default(employee8e0fd50a402c49edbfc8efbdc0402ec1job_title) USING GSI
[2022-09-10 15:32:44,278] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee8e0fd50a402c49edbfc8efbdc0402ec1job_title) USING GSI
[2022-09-10 15:32:44,304] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee8e0fd50a402c49edbfc8efbdc0402ec1job_title%29+USING+GSI
[2022-09-10 15:32:44,332] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.202289ms
[2022-09-10 15:32:45,364] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8e0fd50a402c49edbfc8efbdc0402ec1job_title'
[2022-09-10 15:32:45,391] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8e0fd50a402c49edbfc8efbdc0402ec1job_title%27
[2022-09-10 15:32:45,401] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.139727ms
[2022-09-10 15:32:46,431] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8e0fd50a402c49edbfc8efbdc0402ec1job_title'
[2022-09-10 15:32:46,457] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8e0fd50a402c49edbfc8efbdc0402ec1job_title%27
[2022-09-10 15:32:46,464] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.253364ms
[2022-09-10 15:32:46,904] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 15:32:47,076] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:32:48,062] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:32:48,467] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:32:48,500] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:32:48,526] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 15:32:48,530] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.317176ms
[2022-09-10 15:32:48,530] - [task:3246] INFO - {'requestID': 'a94f02e8-b77c-418a-9f7a-8b969a3cb403', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee8e0fd50a402c49edbfc8efbdc0402ec1job_title', 'index_id': '824144a6ba5e7e13', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.317176ms', 'executionTime': '2.257426ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 15:32:48,531] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:32:48,531] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:32:48,531] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:32:48,532] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:32:48,532] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 15:32:48,532] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:32:48,532] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:32:48,532] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:32:48,533] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:32:49,532] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:32:49,563] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:32:49,592] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 15:32:49,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 129.787758ms
[2022-09-10 15:32:49,732] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:32:49,732] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:32:50,502] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:32:50,503] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:32:51,534] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8e0fd50a402c49edbfc8efbdc0402ec1job_title'
[2022-09-10 15:32:51,560] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8e0fd50a402c49edbfc8efbdc0402ec1job_title%27
[2022-09-10 15:32:51,570] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.515848ms
[2022-09-10 15:32:51,596] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee8e0fd50a402c49edbfc8efbdc0402ec1job_title ON default USING GSI
[2022-09-10 15:32:51,621] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee8e0fd50a402c49edbfc8efbdc0402ec1job_title+ON+default+USING+GSI
[2022-09-10 15:32:51,672] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.407306ms
[2022-09-10 15:32:51,708] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8e0fd50a402c49edbfc8efbdc0402ec1job_title'
[2022-09-10 15:32:51,734] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8e0fd50a402c49edbfc8efbdc0402ec1job_title%27
[2022-09-10 15:32:51,741] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.720358ms
[2022-09-10 15:32:51,742] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ca5c3952-275a-43fa-9e87-3a4e8878a09f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.720358ms', 'executionTime': '5.654537ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:32:51,850] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:32:51,853] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:32:51,853] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:32:52,209] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:52,265] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:32:52,266] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:32:52,327] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:32:52,383] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:32:52,386] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:32:52,386] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:32:52,737] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:52,799] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:32:52,799] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:32:52,863] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:32:52,921] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:32:52,922] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:32:52,922] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:32:52,947] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:32:52,978] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:32:52,987] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.391815ms
[2022-09-10 15:32:53,020] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:32:53,047] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:32:53,095] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.146214ms
[2022-09-10 15:32:53,168] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:32:53,169] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.77062386557216, 'mem_free': 13098663936, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:32:53,169] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:32:53,173] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:32:53,173] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:32:53,543] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:53,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:32:53,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:32:53,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:53,913] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:32:53,913] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:32:54,413] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:54,420] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:32:54,420] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:32:55,012] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:32:59,275] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:32:59,420] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:32:59,970] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:32:59,997] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:32:59,997] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:33:00,055] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:33:00,109] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:33:00,162] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:33:00,163] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:33:00,243] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:33:00,244] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:33:00,270] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:33:00,399] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:33:00,399] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:33:00,426] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:33:00,451] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:33:00,451] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:33:00,478] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:33:00,504] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:33:00,504] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:33:00,535] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:33:00,568] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:33:00,568] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:33:00,595] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:33:00,596] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:33:00,596] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:33:00,596] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 75.928s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_3'}
[2022-09-10 15:33:00,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:00,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:01,015] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:01,047] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:33:01,128] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:33:01,128] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-10 15:33:01,129] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:33:01,725] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:33:01,753] - [task:165] INFO -  {'uptime': '242', 'memoryTotal': 15466930176, 'memoryFree': 13191135232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:33:01,781] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:33:01,781] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:33:01,782] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:33:01,823] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:33:01,859] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:33:01,860] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:33:01,890] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:33:01,891] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:33:01,891] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:33:01,891] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:33:01,943] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:33:01,946] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:01,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:02,285] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:02,286] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:33:02,367] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:33:02,368] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:33:02,399] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:33:02,399] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:33:02,426] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:33:02,513] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:33:02,542] - [task:165] INFO -  {'uptime': '239', 'memoryTotal': 15466930176, 'memoryFree': 13113782272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:33:02,570] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:33:02,603] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:33:02,603] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:33:02,653] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:33:02,656] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:02,656] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:02,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:02,989] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:33:03,074] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:33:03,075] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:33:03,104] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:33:03,105] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:33:03,130] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:33:03,214] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:33:03,240] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13195923456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:33:03,266] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:33:03,294] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:33:03,295] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:33:03,344] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:33:03,349] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:03,350] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:03,653] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:03,654] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:33:03,728] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:33:03,729] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:33:03,758] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:33:03,759] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:33:03,784] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:33:03,871] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:33:03,898] - [task:165] INFO -  {'uptime': '236', 'memoryTotal': 15466930176, 'memoryFree': 13133377536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:33:03,924] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:33:03,954] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:33:03,957] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:33:04,008] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:33:04,011] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:04,011] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:04,340] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:04,341] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:33:04,418] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:33:04,419] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:33:04,450] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:33:04,450] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:33:04,476] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:33:04,535] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:33:05,049] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:33:10,052] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:33:10,140] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:33:10,145] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:10,146] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:10,465] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:10,466] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:33:10,542] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:33:10,543] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:33:10,544] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:33:11,653] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:33:11,710] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 15:33:11,710] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:33:12,601] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:33:12,919] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:33:13,212] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:33:13,215] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:33:13,283] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:13,283] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:13,596] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:13,601] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:13,601] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:13,904] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:13,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:13,910] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:14,281] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:14,289] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:14,289] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:14,878] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:18,887] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:33:18,888] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.42226985503159, 'mem_free': 13195231232, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:33:18,888] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:33:18,888] - [basetestcase:467] INFO - Time to execute basesetup : 18.19475030899048
[2022-09-10 15:33:18,941] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:33:18,941] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:33:18,994] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:33:18,994] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:33:19,050] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:33:19,050] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:33:19,105] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:33:19,106] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:33:19,160] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:33:19,225] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:33:19,225] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:33:19,226] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:33:24,238] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:33:24,242] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:33:24,242] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:33:24,556] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:33:25,495] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:33:25,661] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:33:28,302] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:33:28,317] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 15:33:29,154] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:33:29,155] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:33:29,155] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:33:59,185] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:33:59,215] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:33:59,241] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:33:59,309] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.55726ms
[2022-09-10 15:33:59,309] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '60ff9d38-5c81-49dd-842a-ea0dd7cd7145', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.55726ms', 'executionTime': '66.495929ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:33:59,310] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:33:59,335] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:33:59,361] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:34:00,049] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 685.99707ms
[2022-09-10 15:34:00,050] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:34:00,112] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:34:00,153] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:34:00,167] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 11.601181ms
[2022-09-10 15:34:00,376] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:34:00,412] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:34:00,424] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:34:00,424] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:34:00,440] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:34:00,501] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:34:01,335] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5897cedc659648678015bd609a76f37ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:34:01,361] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5897cedc659648678015bd609a76f37ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:34:01,421] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.28329ms
[2022-09-10 15:34:01,422] - [base_gsi:282] INFO - BUILD INDEX on default(employee5897cedc659648678015bd609a76f37ajob_title) USING GSI
[2022-09-10 15:34:02,454] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5897cedc659648678015bd609a76f37ajob_title) USING GSI
[2022-09-10 15:34:02,481] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee5897cedc659648678015bd609a76f37ajob_title%29+USING+GSI
[2022-09-10 15:34:02,503] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.210888ms
[2022-09-10 15:34:03,534] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5897cedc659648678015bd609a76f37ajob_title'
[2022-09-10 15:34:03,561] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5897cedc659648678015bd609a76f37ajob_title%27
[2022-09-10 15:34:03,573] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.767348ms
[2022-09-10 15:34:04,603] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5897cedc659648678015bd609a76f37ajob_title'
[2022-09-10 15:34:04,628] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5897cedc659648678015bd609a76f37ajob_title%27
[2022-09-10 15:34:04,638] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.404842ms
[2022-09-10 15:34:05,022] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 15:34:05,277] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:34:05,948] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:34:06,641] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:34:06,675] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:34:06,703] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 15:34:06,709] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.988924ms
[2022-09-10 15:34:06,709] - [task:3246] INFO - {'requestID': 'af839ed0-0a55-4cb8-94d3-ae66b233b755', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5897cedc659648678015bd609a76f37ajob_title', 'index_id': '444fecad893985fd', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.988924ms', 'executionTime': '3.890349ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 15:34:06,710] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:34:06,710] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:34:06,710] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:34:06,711] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:34:06,711] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 15:34:06,711] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:34:06,711] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:34:06,712] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:34:06,712] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:34:07,711] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:34:07,742] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:34:07,770] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 15:34:07,896] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 116.521024ms
[2022-09-10 15:34:07,897] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:34:07,897] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:34:08,686] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:34:08,686] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:34:09,717] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5897cedc659648678015bd609a76f37ajob_title'
[2022-09-10 15:34:09,745] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5897cedc659648678015bd609a76f37ajob_title%27
[2022-09-10 15:34:09,758] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 11.083477ms
[2022-09-10 15:34:09,786] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5897cedc659648678015bd609a76f37ajob_title ON default USING GSI
[2022-09-10 15:34:09,812] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5897cedc659648678015bd609a76f37ajob_title+ON+default+USING+GSI
[2022-09-10 15:34:09,855] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.319483ms
[2022-09-10 15:34:09,893] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5897cedc659648678015bd609a76f37ajob_title'
[2022-09-10 15:34:09,920] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5897cedc659648678015bd609a76f37ajob_title%27
[2022-09-10 15:34:09,928] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.664493ms
[2022-09-10 15:34:09,928] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '651aaa46-092d-4a51-8351-f27478477a5a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.664493ms', 'executionTime': '5.594059ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:34:10,038] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:34:10,041] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:10,041] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:10,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:10,486] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:34:10,486] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:34:10,555] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:34:10,611] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:34:10,614] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:10,615] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:11,018] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:11,080] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:34:11,081] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:34:11,158] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:34:11,216] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:34:11,217] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:34:11,217] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:34:11,245] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:34:11,272] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:34:11,280] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.659462ms
[2022-09-10 15:34:11,306] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:34:11,332] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:34:11,385] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.818782ms
[2022-09-10 15:34:11,453] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:34:11,454] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.56644508904037, 'mem_free': 13092413440, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:34:11,454] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:34:11,458] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:11,458] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:11,851] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:11,856] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:11,856] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:12,462] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:12,470] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:12,470] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:13,158] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:13,169] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:13,169] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:13,846] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:17,902] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:34:18,048] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:34:19,212] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:34:19,239] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:34:19,240] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:34:19,300] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:34:19,355] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:34:19,408] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:34:19,409] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:34:19,487] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:34:19,488] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:34:19,516] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:34:19,651] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:34:19,651] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:34:19,680] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:34:19,706] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:34:19,706] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:34:19,732] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:34:19,758] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:34:19,759] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:34:19,785] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:34:19,810] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:34:19,810] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:34:19,836] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:34:19,836] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:34:19,836] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:34:19,837] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.197s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_4'}
[2022-09-10 15:34:19,938] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:19,938] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:20,318] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:20,350] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:34:20,431] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:34:20,432] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-10 15:34:20,432] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:34:20,967] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:34:20,994] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13155336192, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:34:21,020] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:34:21,021] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:34:21,021] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:34:21,050] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:34:21,089] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:34:21,089] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:34:21,117] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:34:21,118] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:34:21,118] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:34:21,119] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:34:21,166] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:34:21,169] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:21,169] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:21,524] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:21,525] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:34:21,606] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:34:21,607] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:34:21,636] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:34:21,637] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:34:21,662] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:34:21,748] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:34:21,776] - [task:165] INFO -  {'uptime': '320', 'memoryTotal': 15466930176, 'memoryFree': 13157552128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:34:21,803] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:34:21,832] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:34:21,832] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:34:21,882] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:34:21,885] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:21,885] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:22,239] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:22,240] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:34:22,320] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:34:22,322] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:34:22,350] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:34:22,350] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:34:22,376] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:34:22,458] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:34:22,484] - [task:165] INFO -  {'uptime': '317', 'memoryTotal': 15466930176, 'memoryFree': 13157761024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:34:22,509] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:34:22,537] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:34:22,537] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:34:22,586] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:34:22,590] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:22,591] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:22,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:22,973] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:34:23,063] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:34:23,064] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:34:23,094] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:34:23,095] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:34:23,125] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:34:23,214] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:34:23,242] - [task:165] INFO -  {'uptime': '316', 'memoryTotal': 15466930176, 'memoryFree': 13157859328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:34:23,268] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:34:23,295] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:34:23,296] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:34:23,353] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:34:23,356] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:23,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:23,735] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:23,737] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:34:23,821] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:34:23,823] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:34:23,853] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:34:23,854] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:34:23,881] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:34:23,936] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:34:24,479] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:34:29,484] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:34:29,573] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:34:29,577] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:29,578] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:29,993] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:29,994] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:34:30,082] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:34:30,083] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:34:30,083] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:34:31,090] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:34:31,145] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 15:34:31,146] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:34:32,062] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:34:32,386] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:34:32,647] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:34:32,649] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:34:32,703] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:32,703] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:33,068] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:33,073] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:33,073] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:33,436] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:33,440] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:33,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:33,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:33,920] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:33,920] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:34,562] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:38,975] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:34:38,975] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.03487618667211, 'mem_free': 13159698432, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:34:38,975] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:34:38,976] - [basetestcase:467] INFO - Time to execute basesetup : 19.040371894836426
[2022-09-10 15:34:39,026] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:34:39,026] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:34:39,079] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:34:39,079] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:34:39,135] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:34:39,135] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:34:39,191] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:34:39,191] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:34:39,242] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:34:39,300] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:34:39,301] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:34:39,302] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:34:44,312] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:34:44,315] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:34:44,315] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:34:44,696] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:34:45,667] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:34:45,912] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:34:48,425] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:34:48,445] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 15:34:49,127] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:34:49,128] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:34:49,128] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:35:19,157] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:35:19,186] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:35:19,210] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:35:19,276] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.794471ms
[2022-09-10 15:35:19,277] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f0a47cf1-eef1-450e-abed-77acc79ebd11', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.794471ms', 'executionTime': '64.728911ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:35:19,277] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:35:19,302] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:35:19,327] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:35:20,043] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 713.650895ms
[2022-09-10 15:35:20,043] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:35:20,094] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:35:20,134] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:35:20,145] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.001029ms
[2022-09-10 15:35:20,381] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:35:20,417] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:35:20,432] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:35:20,435] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:35:20,447] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:35:20,515] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:35:21,345] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec6968b4c29224e3fa50eb01970b270d5job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:35:21,370] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec6968b4c29224e3fa50eb01970b270d5job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:35:21,413] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.297287ms
[2022-09-10 15:35:21,413] - [base_gsi:282] INFO - BUILD INDEX on default(employeec6968b4c29224e3fa50eb01970b270d5job_title) USING GSI
[2022-09-10 15:35:22,444] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec6968b4c29224e3fa50eb01970b270d5job_title) USING GSI
[2022-09-10 15:35:22,470] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec6968b4c29224e3fa50eb01970b270d5job_title%29+USING+GSI
[2022-09-10 15:35:22,500] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.743451ms
[2022-09-10 15:35:23,531] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec6968b4c29224e3fa50eb01970b270d5job_title'
[2022-09-10 15:35:23,559] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec6968b4c29224e3fa50eb01970b270d5job_title%27
[2022-09-10 15:35:23,569] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.633131ms
[2022-09-10 15:35:24,600] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec6968b4c29224e3fa50eb01970b270d5job_title'
[2022-09-10 15:35:24,625] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec6968b4c29224e3fa50eb01970b270d5job_title%27
[2022-09-10 15:35:24,634] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.625136ms
[2022-09-10 15:35:25,117] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 15:35:25,283] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:35:26,163] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:35:26,352] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-10 15:35:26,352] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:35:26,532] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-10 15:35:26,533] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-10 15:35:26,533] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2151543623, 0, b'')
[2022-09-10 15:35:26,638] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:35:26,667] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:35:26,695] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 15:35:26,700] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.916686ms
[2022-09-10 15:35:26,700] - [task:3246] INFO - {'requestID': 'b60f4d8b-a628-4c27-a406-f868ce99f6ae', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec6968b4c29224e3fa50eb01970b270d5job_title', 'index_id': 'e17d3274ea5092f', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.916686ms', 'executionTime': '2.851843ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-09-10 15:35:26,700] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:35:26,701] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:35:26,701] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:35:26,701] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:35:26,702] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 15:35:26,702] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:35:26,702] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:35:26,702] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:35:26,702] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:35:27,702] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:35:27,732] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:35:27,758] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 15:35:27,880] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 113.058581ms
[2022-09-10 15:35:27,880] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:35:27,880] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:35:28,657] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:35:28,657] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:35:29,689] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec6968b4c29224e3fa50eb01970b270d5job_title'
[2022-09-10 15:35:29,716] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec6968b4c29224e3fa50eb01970b270d5job_title%27
[2022-09-10 15:35:29,726] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.618476ms
[2022-09-10 15:35:29,754] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec6968b4c29224e3fa50eb01970b270d5job_title ON default USING GSI
[2022-09-10 15:35:29,780] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec6968b4c29224e3fa50eb01970b270d5job_title+ON+default+USING+GSI
[2022-09-10 15:35:29,839] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.842423ms
[2022-09-10 15:35:29,876] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec6968b4c29224e3fa50eb01970b270d5job_title'
[2022-09-10 15:35:29,906] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec6968b4c29224e3fa50eb01970b270d5job_title%27
[2022-09-10 15:35:29,915] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.471294ms
[2022-09-10 15:35:29,916] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '990cc79c-8bff-46c0-bcb2-0ab210948ed3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.471294ms', 'executionTime': '7.406712ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:35:30,025] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:35:30,028] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:30,028] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:30,478] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:30,534] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:35:30,534] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:35:30,611] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:35:30,666] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:35:30,669] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:30,670] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:31,116] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:31,186] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:35:31,187] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:35:31,269] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:35:31,325] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:35:31,326] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:35:31,326] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:35:31,352] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:35:31,378] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:35:31,385] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.635562ms
[2022-09-10 15:35:31,412] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:35:31,438] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:35:31,480] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.103249ms
[2022-09-10 15:35:31,554] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:35:31,554] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.931796621789111, 'mem_free': 13039218688, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:35:31,555] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:35:31,561] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:31,561] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:32,007] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:32,012] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:32,012] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:32,648] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:32,656] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:32,656] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:33,421] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:33,429] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:33,429] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:34,215] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:39,056] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:35:39,202] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:35:39,623] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:35:39,651] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:35:39,651] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:35:39,707] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:35:39,760] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:35:39,812] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:35:39,813] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:35:39,896] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:35:39,897] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:35:39,928] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:35:40,069] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:35:40,069] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:35:40,095] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:35:40,121] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:35:40,121] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:35:40,149] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:35:40,174] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:35:40,174] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:35:40,201] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:35:40,226] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:35:40,227] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:35:40,254] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:35:40,254] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:35:40,254] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:35:40,255] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.372s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_5'}
[2022-09-10 15:35:40,366] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:40,367] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:40,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:40,807] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:35:40,887] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:35:40,887] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-10 15:35:40,888] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:35:41,393] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:35:41,420] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13119574016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:35:41,446] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:35:41,446] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:35:41,447] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:35:41,477] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:35:41,515] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:35:41,516] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:35:41,544] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:35:41,545] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:35:41,546] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:35:41,546] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:35:41,593] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:35:41,596] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:41,596] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:42,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:42,009] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:35:42,096] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:35:42,097] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:35:42,127] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:35:42,128] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:35:42,154] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:35:42,237] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:35:42,263] - [task:165] INFO -  {'uptime': '400', 'memoryTotal': 15466930176, 'memoryFree': 13118701568, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:35:42,289] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:35:42,316] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:35:42,316] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:35:42,366] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:35:42,368] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:42,369] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:42,821] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:42,822] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:35:42,918] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:35:42,919] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:35:42,949] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:35:42,949] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:35:42,976] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:35:43,066] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:35:43,095] - [task:165] INFO -  {'uptime': '397', 'memoryTotal': 15466930176, 'memoryFree': 13119504384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:35:43,123] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:35:43,155] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:35:43,155] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:35:43,207] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:35:43,211] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:43,211] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:43,617] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:43,619] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:35:43,711] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:35:43,712] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:35:43,742] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:35:43,742] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:35:43,768] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:35:43,850] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:35:43,877] - [task:165] INFO -  {'uptime': '396', 'memoryTotal': 15466930176, 'memoryFree': 13119938560, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:35:43,902] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:35:43,930] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:35:43,930] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:35:43,982] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:35:43,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:43,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:44,393] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:44,395] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:35:44,482] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:35:44,484] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:35:44,513] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:35:44,513] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:35:44,538] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:35:44,595] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:35:45,271] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:35:50,272] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:35:50,356] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:35:50,360] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:50,360] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:50,798] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:50,799] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:35:50,891] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:35:50,893] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:35:50,893] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:35:51,719] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:35:51,773] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 15:35:51,774] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:35:52,668] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:35:52,983] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:35:53,260] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:35:53,262] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:35:53,323] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:53,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:53,762] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:53,766] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:53,766] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:54,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:54,180] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:54,180] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:54,861] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:35:54,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:35:54,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:35:55,621] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:36:00,849] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:36:00,850] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.77229017972139, 'mem_free': 13123674112, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:36:00,850] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:36:00,850] - [basetestcase:467] INFO - Time to execute basesetup : 20.48604154586792
[2022-09-10 15:36:00,906] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:36:00,906] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:36:00,963] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:36:00,964] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:36:01,020] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:36:01,020] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:36:01,078] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:36:01,079] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:36:01,134] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:36:01,199] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:36:01,199] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:36:01,199] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:36:06,211] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:36:06,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:36:06,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:36:06,625] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:36:07,601] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:36:07,852] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:36:10,327] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:36:10,343] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 15:36:11,387] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:36:11,388] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:36:11,388] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:36:41,391] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:36:41,420] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:36:41,446] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:36:41,516] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.045842ms
[2022-09-10 15:36:41,516] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a453b78d-926e-495f-85f9-3bc6015fb0f7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.045842ms', 'executionTime': '67.978846ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:36:41,516] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:36:41,543] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:36:41,569] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:36:42,265] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 693.644493ms
[2022-09-10 15:36:42,265] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:36:42,326] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:36:42,364] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:36:42,372] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.176027ms
[2022-09-10 15:36:42,577] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:36:42,611] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:36:42,625] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:36:42,625] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:36:42,642] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:36:42,721] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:36:43,532] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed5908b8f6d8a4337ae0b9f303c21a854job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:36:43,557] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed5908b8f6d8a4337ae0b9f303c21a854job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:36:43,631] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 72.041021ms
[2022-09-10 15:36:43,632] - [base_gsi:282] INFO - BUILD INDEX on default(employeed5908b8f6d8a4337ae0b9f303c21a854job_title) USING GSI
[2022-09-10 15:36:44,663] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed5908b8f6d8a4337ae0b9f303c21a854job_title) USING GSI
[2022-09-10 15:36:44,690] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed5908b8f6d8a4337ae0b9f303c21a854job_title%29+USING+GSI
[2022-09-10 15:36:44,713] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.843095ms
[2022-09-10 15:36:45,746] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed5908b8f6d8a4337ae0b9f303c21a854job_title'
[2022-09-10 15:36:45,772] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed5908b8f6d8a4337ae0b9f303c21a854job_title%27
[2022-09-10 15:36:45,782] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.409591ms
[2022-09-10 15:36:46,814] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed5908b8f6d8a4337ae0b9f303c21a854job_title'
[2022-09-10 15:36:46,840] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed5908b8f6d8a4337ae0b9f303c21a854job_title%27
[2022-09-10 15:36:46,847] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.511883ms
[2022-09-10 15:36:47,336] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:36:47,506] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:36:51,153] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:36:51,853] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:36:51,886] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:36:51,911] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 15:36:51,915] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.16726ms
[2022-09-10 15:36:51,915] - [task:3246] INFO - {'requestID': 'a4ff4d1b-a46d-4ef4-97e8-58c819d36319', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeed5908b8f6d8a4337ae0b9f303c21a854job_title', 'index_id': '6bc2e1aa551d8ef7', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.16726ms', 'executionTime': '2.108001ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 15:36:51,916] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:36:51,916] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:36:51,916] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:36:51,917] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:36:51,917] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 15:36:51,917] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:36:51,917] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:36:51,918] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:36:51,918] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:36:52,917] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:36:52,946] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:36:52,971] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 15:36:53,169] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 182.357957ms
[2022-09-10 15:36:53,169] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:36:53,170] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:36:54,744] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:36:54,745] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:36:55,776] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed5908b8f6d8a4337ae0b9f303c21a854job_title'
[2022-09-10 15:36:55,802] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed5908b8f6d8a4337ae0b9f303c21a854job_title%27
[2022-09-10 15:36:55,813] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.695035ms
[2022-09-10 15:36:55,840] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed5908b8f6d8a4337ae0b9f303c21a854job_title ON default USING GSI
[2022-09-10 15:36:55,867] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed5908b8f6d8a4337ae0b9f303c21a854job_title+ON+default+USING+GSI
[2022-09-10 15:36:55,931] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.952258ms
[2022-09-10 15:36:55,970] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed5908b8f6d8a4337ae0b9f303c21a854job_title'
[2022-09-10 15:36:55,998] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed5908b8f6d8a4337ae0b9f303c21a854job_title%27
[2022-09-10 15:36:56,008] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.760736ms
[2022-09-10 15:36:56,008] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ef79f460-9a1c-4d0a-a846-d2b84398fe57', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.760736ms', 'executionTime': '7.690794ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:36:56,112] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:36:56,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:36:56,115] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:36:56,653] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:36:56,707] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:36:56,708] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:36:56,792] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:36:56,850] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:36:56,852] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:36:56,853] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:36:57,410] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:36:57,470] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:36:57,470] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:36:57,563] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:36:57,622] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:36:57,622] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:36:57,622] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:36:57,649] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:36:57,676] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:36:57,687] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.045696ms
[2022-09-10 15:36:57,713] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:36:57,738] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:36:57,782] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.678757ms
[2022-09-10 15:36:57,856] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:36:57,857] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.91553161795011, 'mem_free': 12942581760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:36:57,857] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:36:57,861] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:36:57,861] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:36:58,423] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:36:58,428] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:36:58,428] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:36:58,980] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:36:58,985] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:36:58,985] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:36:59,899] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:36:59,907] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:36:59,907] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:00,848] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:06,877] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:37:07,017] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:37:08,309] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:37:08,337] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:37:08,337] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:37:08,399] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:37:08,452] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:37:08,504] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:37:08,505] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:37:08,584] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:37:08,585] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:37:08,610] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:37:08,736] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:37:08,737] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:37:08,763] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:37:08,788] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:37:08,788] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:37:08,814] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:37:08,840] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:37:08,840] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:37:08,867] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:37:08,893] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:37:08,893] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:37:08,920] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:37:08,920] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:37:08,920] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:37:08,920] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 88.608s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_6'}
[2022-09-10 15:37:09,027] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:09,028] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:09,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:09,583] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:37:09,665] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:37:09,665] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-10 15:37:09,666] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:37:10,055] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:37:10,082] - [task:165] INFO -  {'uptime': '491', 'memoryTotal': 15466930176, 'memoryFree': 13058084864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:37:10,108] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:37:10,108] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:37:10,109] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:37:10,148] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:37:10,184] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:37:10,184] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:37:10,215] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:37:10,216] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:37:10,216] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:37:10,216] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:37:10,272] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:37:10,275] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:10,276] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:10,787] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:10,788] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:37:10,888] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:37:10,889] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:37:10,918] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:37:10,919] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:37:10,945] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:37:11,028] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:37:11,054] - [task:165] INFO -  {'uptime': '490', 'memoryTotal': 15466930176, 'memoryFree': 13058506752, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:37:11,079] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:37:11,106] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:37:11,107] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:37:11,158] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:37:11,161] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:11,161] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:11,674] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:11,675] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:37:11,773] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:37:11,774] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:37:11,804] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:37:11,805] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:37:11,831] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:37:11,913] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:37:11,940] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13057617920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:37:11,966] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:37:11,993] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:37:11,993] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:37:12,044] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:37:12,048] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:12,048] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:12,543] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:12,544] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:37:12,646] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:37:12,647] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:37:12,676] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:37:12,677] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:37:12,703] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:37:12,788] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:37:12,815] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13058846720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:37:12,840] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:37:12,867] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:37:12,867] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:37:12,917] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:37:12,920] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:12,921] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:13,425] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:13,426] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:37:13,525] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:37:13,526] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:37:13,556] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:37:13,557] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:37:13,582] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:37:13,637] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:37:14,169] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:37:19,171] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:37:19,253] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:37:19,258] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:19,258] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:19,770] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:19,771] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:37:19,870] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:37:19,871] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:37:19,871] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:37:20,791] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:37:20,845] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 15:37:20,846] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:37:21,755] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:37:22,057] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:37:22,310] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:37:22,312] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:37:22,366] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:22,366] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:22,860] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:22,866] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:22,866] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:23,416] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:23,423] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:23,423] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:24,321] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:24,329] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:24,329] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:25,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:30,836] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:37:30,836] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.83534302547797, 'mem_free': 13061685248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:37:30,836] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:37:30,836] - [basetestcase:467] INFO - Time to execute basesetup : 21.811161756515503
[2022-09-10 15:37:30,887] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:37:30,887] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:37:30,943] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:37:30,943] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:37:30,999] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:37:30,999] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:37:31,054] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:37:31,055] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:37:31,106] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:37:31,170] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:37:31,170] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:37:31,171] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:37:36,181] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:37:36,185] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:37:36,185] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:37:36,715] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:37:37,708] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:37:37,968] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:37:40,302] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:37:40,319] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 15:37:41,705] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:37:41,705] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:37:41,705] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:38:11,727] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:38:11,757] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:38:11,782] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:38:11,850] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.979741ms
[2022-09-10 15:38:11,851] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'de53410c-c9c3-472a-bd45-99b3e59e9cc0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.979741ms', 'executionTime': '65.908008ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:38:11,851] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:38:11,877] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:38:11,903] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:38:12,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 699.332585ms
[2022-09-10 15:38:12,604] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:38:12,688] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:38:12,726] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:38:12,735] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.77021ms
[2022-09-10 15:38:12,934] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:38:12,964] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:38:12,981] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:38:12,981] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:38:12,995] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:38:13,072] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:38:13,898] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5956b97e005f4b489e0b3f1f5ab3881ejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:38:13,925] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5956b97e005f4b489e0b3f1f5ab3881ejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:38:13,988] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.889426ms
[2022-09-10 15:38:13,989] - [base_gsi:282] INFO - BUILD INDEX on default(employee5956b97e005f4b489e0b3f1f5ab3881ejob_title) USING GSI
[2022-09-10 15:38:15,020] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5956b97e005f4b489e0b3f1f5ab3881ejob_title) USING GSI
[2022-09-10 15:38:15,048] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee5956b97e005f4b489e0b3f1f5ab3881ejob_title%29+USING+GSI
[2022-09-10 15:38:15,072] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.661822ms
[2022-09-10 15:38:16,104] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5956b97e005f4b489e0b3f1f5ab3881ejob_title'
[2022-09-10 15:38:16,130] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5956b97e005f4b489e0b3f1f5ab3881ejob_title%27
[2022-09-10 15:38:16,140] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.823339ms
[2022-09-10 15:38:17,175] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5956b97e005f4b489e0b3f1f5ab3881ejob_title'
[2022-09-10 15:38:17,202] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5956b97e005f4b489e0b3f1f5ab3881ejob_title%27
[2022-09-10 15:38:17,209] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.357106ms
[2022-09-10 15:38:17,697] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 15:38:17,865] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:38:18,968] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:38:19,579] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 15:38:19,750] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:38:21,052] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:38:21,446] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:38:21,616] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:38:25,289] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:38:26,231] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:38:26,266] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:38:26,293] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 15:38:26,297] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.990965ms
[2022-09-10 15:38:26,298] - [task:3246] INFO - {'requestID': 'fc383088-4384-4407-b39b-585a7ab97639', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5956b97e005f4b489e0b3f1f5ab3881ejob_title', 'index_id': '4c2b51f5f909dbf1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.990965ms', 'executionTime': '2.930777ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 15:38:26,298] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:38:26,298] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:38:26,299] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:38:26,299] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:38:26,299] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 15:38:26,300] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:38:26,300] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:38:26,300] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:38:26,300] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 15:38:27,300] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:38:27,330] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 15:38:27,356] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 15:38:27,735] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 159.08495ms
[2022-09-10 15:38:27,735] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:38:27,736] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:38:29,274] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:38:29,275] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 15:38:30,306] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5956b97e005f4b489e0b3f1f5ab3881ejob_title'
[2022-09-10 15:38:30,332] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5956b97e005f4b489e0b3f1f5ab3881ejob_title%27
[2022-09-10 15:38:30,342] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.984548ms
[2022-09-10 15:38:30,368] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5956b97e005f4b489e0b3f1f5ab3881ejob_title ON default USING GSI
[2022-09-10 15:38:30,393] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5956b97e005f4b489e0b3f1f5ab3881ejob_title+ON+default+USING+GSI
[2022-09-10 15:38:30,454] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.666105ms
[2022-09-10 15:38:30,488] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5956b97e005f4b489e0b3f1f5ab3881ejob_title'
[2022-09-10 15:38:30,517] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5956b97e005f4b489e0b3f1f5ab3881ejob_title%27
[2022-09-10 15:38:30,525] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.948531ms
[2022-09-10 15:38:30,525] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e48b7d32-0940-4eb8-8381-6402102e0ad4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.948531ms', 'executionTime': '5.876547ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:38:30,638] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:38:30,642] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:30,642] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:31,312] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:31,367] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:38:31,368] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:38:31,470] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:38:31,527] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:38:31,530] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:31,530] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:32,162] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:32,240] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:38:32,240] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:38:32,347] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:38:32,406] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:38:32,406] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:38:32,406] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:38:32,432] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:38:32,458] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:38:32,467] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.012127ms
[2022-09-10 15:38:32,495] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:38:32,520] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:38:32,568] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.129629ms
[2022-09-10 15:38:32,642] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:38:32,643] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.33676156123096, 'mem_free': 12866809856, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:38:32,643] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:38:32,647] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:32,647] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:33,304] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:33,309] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:33,309] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:33,982] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:33,987] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:33,987] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:34,986] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:34,993] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:34,993] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:36,109] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:43,250] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:38:43,390] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:38:44,336] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:38:44,364] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:38:44,364] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:38:44,421] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:38:44,474] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:38:44,537] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:38:44,538] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:38:44,617] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:38:44,618] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:38:44,643] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:38:44,771] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:38:44,772] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:38:44,797] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:38:44,823] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:38:44,823] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:38:44,850] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:38:44,875] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:38:44,875] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:38:44,901] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:38:44,926] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:38:44,926] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:38:44,953] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:38:44,953] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 15:38:44,954] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:38:44,954] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 95.979s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_7'}
[2022-09-10 15:38:45,086] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:45,086] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:45,664] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:45,697] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:38:45,780] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:38:45,780] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-10 15:38:45,780] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:38:46,112] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:38:46,139] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 13003210752, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:38:46,164] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:38:46,164] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:38:46,164] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:38:46,195] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:38:46,230] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:38:46,231] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:38:46,258] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:38:46,259] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:38:46,259] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:38:46,259] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:38:46,307] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:38:46,310] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:46,310] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:46,893] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:46,894] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:38:47,007] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:38:47,008] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:38:47,037] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:38:47,038] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:38:47,063] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:38:47,153] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:38:47,182] - [task:165] INFO -  {'uptime': '586', 'memoryTotal': 15466930176, 'memoryFree': 13002518528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:38:47,208] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:38:47,235] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:38:47,235] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:38:47,286] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:38:47,289] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:47,289] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:47,899] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:47,900] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:38:48,022] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:38:48,023] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:38:48,054] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:38:48,055] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:38:48,081] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:38:48,167] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:38:48,196] - [task:165] INFO -  {'uptime': '583', 'memoryTotal': 15466930176, 'memoryFree': 13001764864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:38:48,223] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:38:48,254] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:38:48,254] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:38:48,305] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:38:48,310] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:48,310] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:48,926] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:48,927] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:38:49,047] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:38:49,048] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:38:49,077] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:38:49,078] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:38:49,103] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:38:49,190] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:38:49,216] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 13002194944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:38:49,241] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:38:49,268] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:38:49,269] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:38:49,319] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:38:49,322] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:49,322] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:49,958] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:49,959] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:38:50,083] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:38:50,084] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:38:50,115] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:38:50,116] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:38:50,141] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:38:50,197] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:38:50,848] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:38:55,853] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:38:55,938] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:38:55,942] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:55,942] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:56,561] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:56,562] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:38:56,684] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:38:56,685] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:38:56,686] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:38:57,319] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:38:57,374] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 15:38:57,375] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:38:58,296] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:38:58,617] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:38:58,893] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:38:58,897] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 15:38:58,956] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:58,956] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:38:59,561] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:38:59,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:38:59,566] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:00,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:00,429] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:00,429] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:01,504] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:01,518] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:01,518] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:02,614] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:09,042] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:39:09,043] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.61789236178672, 'mem_free': 13022519296, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:39:09,043] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:39:09,043] - [basetestcase:467] INFO - Time to execute basesetup : 23.95970892906189
[2022-09-10 15:39:09,094] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:39:09,094] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:39:09,150] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:39:09,150] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:39:09,203] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:39:09,203] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:39:09,256] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:39:09,256] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:39:09,306] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:39:09,364] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:39:09,365] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:39:09,365] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:39:14,377] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:39:14,380] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:14,380] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:14,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:16,017] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:39:16,183] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:39:18,449] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:39:18,532] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:39:18,532] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:39:18,533] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:39:48,557] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:39:48,587] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:39:48,615] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:39:48,683] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.841944ms
[2022-09-10 15:39:48,684] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b19bab64-19c6-4f86-a837-764d678aaa5c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.841944ms', 'executionTime': '66.772184ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:39:48,684] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:39:48,710] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:39:48,734] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:39:49,392] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 655.745326ms
[2022-09-10 15:39:49,392] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:39:49,448] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:39:49,486] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:39:49,494] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.716036ms
[2022-09-10 15:39:49,690] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:39:49,729] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:39:49,751] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:39:49,752] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:39:49,774] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:39:49,775] - [base_gsi:326] INFO - []
[2022-09-10 15:39:50,656] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee70b63697668a444d8021a4f1d65961b6job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:39:50,681] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee70b63697668a444d8021a4f1d65961b6job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:39:50,735] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.236928ms
[2022-09-10 15:39:50,770] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee70b63697668a444d8021a4f1d65961b6join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:39:50,795] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee70b63697668a444d8021a4f1d65961b6join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:39:50,855] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.309425ms
[2022-09-10 15:39:50,855] - [base_gsi:282] INFO - BUILD INDEX on default(employee70b63697668a444d8021a4f1d65961b6job_title,employee70b63697668a444d8021a4f1d65961b6join_yr) USING GSI
[2022-09-10 15:39:51,886] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee70b63697668a444d8021a4f1d65961b6job_title,employee70b63697668a444d8021a4f1d65961b6join_yr) USING GSI
[2022-09-10 15:39:51,911] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee70b63697668a444d8021a4f1d65961b6job_title%2Cemployee70b63697668a444d8021a4f1d65961b6join_yr%29+USING+GSI
[2022-09-10 15:39:51,954] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.674761ms
[2022-09-10 15:39:52,985] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee70b63697668a444d8021a4f1d65961b6job_title'
[2022-09-10 15:39:53,013] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee70b63697668a444d8021a4f1d65961b6job_title%27
[2022-09-10 15:39:53,023] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.803308ms
[2022-09-10 15:39:54,052] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee70b63697668a444d8021a4f1d65961b6job_title'
[2022-09-10 15:39:54,078] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee70b63697668a444d8021a4f1d65961b6job_title%27
[2022-09-10 15:39:54,087] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.416872ms
[2022-09-10 15:39:54,115] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee70b63697668a444d8021a4f1d65961b6join_yr'
[2022-09-10 15:39:54,140] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee70b63697668a444d8021a4f1d65961b6join_yr%27
[2022-09-10 15:39:54,143] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.522828ms
[2022-09-10 15:39:55,175] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee70b63697668a444d8021a4f1d65961b6job_title'
[2022-09-10 15:39:55,201] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee70b63697668a444d8021a4f1d65961b6job_title%27
[2022-09-10 15:39:55,211] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.247127ms
[2022-09-10 15:39:55,237] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee70b63697668a444d8021a4f1d65961b6job_title ON default USING GSI
[2022-09-10 15:39:55,263] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee70b63697668a444d8021a4f1d65961b6job_title+ON+default+USING+GSI
[2022-09-10 15:39:55,318] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.866386ms
[2022-09-10 15:39:55,355] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee70b63697668a444d8021a4f1d65961b6join_yr'
[2022-09-10 15:39:55,381] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee70b63697668a444d8021a4f1d65961b6join_yr%27
[2022-09-10 15:39:55,389] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.701528ms
[2022-09-10 15:39:55,417] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee70b63697668a444d8021a4f1d65961b6join_yr ON default USING GSI
[2022-09-10 15:39:55,444] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee70b63697668a444d8021a4f1d65961b6join_yr+ON+default+USING+GSI
[2022-09-10 15:39:55,487] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.646811ms
[2022-09-10 15:39:55,524] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee70b63697668a444d8021a4f1d65961b6job_title'
[2022-09-10 15:39:55,551] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee70b63697668a444d8021a4f1d65961b6job_title%27
[2022-09-10 15:39:55,558] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.34738ms
[2022-09-10 15:39:55,558] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '55d7d15e-a398-4d75-99a3-8b3a6bf639c4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.34738ms', 'executionTime': '5.281428ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:39:55,587] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee70b63697668a444d8021a4f1d65961b6join_yr'
[2022-09-10 15:39:55,614] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee70b63697668a444d8021a4f1d65961b6join_yr%27
[2022-09-10 15:39:55,617] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.577331ms
[2022-09-10 15:39:55,617] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b22d48ea-b740-42b1-99bb-a2c8121b003b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.577331ms', 'executionTime': '1.493596ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:39:55,726] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:39:55,730] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:55,730] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:56,396] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:56,452] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:39:56,453] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:39:56,555] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:39:56,614] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:39:56,618] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:56,618] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:57,237] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:57,293] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:39:57,293] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:39:57,399] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:39:57,457] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:39:57,458] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:39:57,458] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:39:57,485] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:39:57,511] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:39:57,521] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.278384ms
[2022-09-10 15:39:57,548] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:39:57,574] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:39:57,624] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.099878ms
[2022-09-10 15:39:57,698] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:39:57,699] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.645825572856911, 'mem_free': 12908507136, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:39:57,699] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:39:57,703] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:57,704] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:58,362] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:58,367] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:58,368] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:39:59,010] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:39:59,015] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:39:59,016] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:00,015] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:00,024] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:00,024] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:01,137] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:08,334] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-10 15:40:08,479] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:40:08,867] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:40:08,895] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:40:08,895] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:40:08,954] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:40:09,009] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:40:09,064] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:40:09,065] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:40:09,149] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:40:09,151] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:40:09,178] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:40:09,312] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:40:09,312] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:40:09,340] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:40:09,368] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:40:09,368] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:40:09,396] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:40:09,423] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:40:09,423] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:40:09,451] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:40:09,477] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:40:09,477] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:40:09,504] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:40:09,505] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 15:40:09,505] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:40:09,505] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.480s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_8'}
[2022-09-10 15:40:09,598] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:09,599] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:10,239] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:10,272] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:40:10,351] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:40:10,352] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-10 15:40:10,352] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:40:10,704] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:40:10,731] - [task:165] INFO -  {'uptime': '671', 'memoryTotal': 15466930176, 'memoryFree': 13016104960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:40:10,756] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:40:10,757] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:40:10,757] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:40:10,793] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:40:10,829] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:40:10,829] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:40:10,857] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:40:10,858] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:40:10,858] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:40:10,858] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:40:10,907] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:40:10,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:10,910] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:11,529] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:11,531] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:40:11,645] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:40:11,646] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:40:11,677] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:40:11,678] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:40:11,705] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:40:11,789] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:40:11,816] - [task:165] INFO -  {'uptime': '671', 'memoryTotal': 15466930176, 'memoryFree': 13015875584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:40:11,841] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:40:11,870] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:40:11,870] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:40:11,921] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:40:11,924] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:11,924] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:12,521] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:12,522] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:40:12,642] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:40:12,643] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:40:12,673] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:40:12,674] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:40:12,700] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:40:12,788] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:40:12,816] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13015990272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:40:12,843] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:40:12,871] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:40:12,872] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:40:12,924] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:40:12,929] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:12,929] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:13,589] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:13,591] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:40:13,713] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:40:13,714] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:40:13,746] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:40:13,748] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:40:13,776] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:40:13,861] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:40:13,892] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13016539136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:40:13,917] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:40:13,948] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:40:13,949] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:40:14,002] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:40:14,005] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:14,005] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:14,592] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:14,593] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:40:14,711] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:40:14,712] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:40:14,742] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:40:14,743] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:40:14,769] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:40:14,830] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:40:15,344] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:40:20,350] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:40:20,435] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:40:20,440] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:20,440] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:21,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:21,057] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:40:21,181] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:40:21,182] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:40:21,183] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:40:21,967] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:40:22,021] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 15:40:22,024] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:40:23,092] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:40:23,407] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:40:23,712] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:40:23,715] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 15:40:23,770] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:23,771] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:24,428] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:24,433] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:24,433] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:25,337] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:25,346] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:25,346] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:26,448] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:26,459] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:26,460] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:27,559] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:34,019] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:40:34,019] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.80306724712531, 'mem_free': 13012803584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:40:34,020] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:40:34,020] - [basetestcase:467] INFO - Time to execute basesetup : 24.426215887069702
[2022-09-10 15:40:34,071] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:40:34,072] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:40:34,125] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:40:34,125] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:40:34,179] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:40:34,179] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:40:34,231] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:40:34,231] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:40:34,281] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:40:34,341] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:40:34,341] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:40:34,342] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:40:39,352] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:40:39,355] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:40:39,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:40:39,958] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:40:40,977] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:40:41,264] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:40:43,415] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:40:43,493] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:40:43,493] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:40:43,493] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:41:13,523] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:41:13,553] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:41:13,579] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:41:13,648] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.780348ms
[2022-09-10 15:41:13,649] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd25ee5b6-ca1f-4dbc-969a-83c80b9d9606', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.780348ms', 'executionTime': '66.705748ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:41:13,649] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:41:13,676] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:41:13,701] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:41:14,382] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 678.523886ms
[2022-09-10 15:41:14,382] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:41:14,439] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:41:14,480] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:41:14,489] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.374319ms
[2022-09-10 15:41:14,701] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:41:14,742] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:41:14,761] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:41:14,762] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:41:14,778] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:41:14,779] - [base_gsi:326] INFO - []
[2022-09-10 15:41:15,658] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:41:15,686] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:41:15,737] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.286062ms
[2022-09-10 15:41:15,738] - [base_gsi:282] INFO - BUILD INDEX on default(employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr) USING GSI
[2022-09-10 15:41:16,770] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr) USING GSI
[2022-09-10 15:41:16,804] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr%29+USING+GSI
[2022-09-10 15:41:16,826] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.197461ms
[2022-09-10 15:41:17,857] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr'
[2022-09-10 15:41:17,882] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr%27
[2022-09-10 15:41:17,892] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.208818ms
[2022-09-10 15:41:18,926] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr'
[2022-09-10 15:41:18,955] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr%27
[2022-09-10 15:41:18,965] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.761311ms
[2022-09-10 15:41:19,997] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr'
[2022-09-10 15:41:20,023] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr%27
[2022-09-10 15:41:20,033] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.377161ms
[2022-09-10 15:41:20,058] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr ON default USING GSI
[2022-09-10 15:41:20,086] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr+ON+default+USING+GSI
[2022-09-10 15:41:20,136] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.3467ms
[2022-09-10 15:41:20,172] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr'
[2022-09-10 15:41:20,199] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee55ea39ea76447f0a54afe50785277a5job_title_join_yr%27
[2022-09-10 15:41:20,206] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.915625ms
[2022-09-10 15:41:20,206] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0411d44a-90d5-4ac2-8944-01381f19bbfe', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.915625ms', 'executionTime': '5.842918ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:41:20,312] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:41:20,315] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:20,316] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:20,931] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:20,987] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:41:20,987] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:41:21,087] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:41:21,142] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:41:21,145] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:21,145] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:21,784] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:21,843] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:41:21,844] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:41:21,947] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:41:22,002] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:41:22,002] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:41:22,003] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:41:22,028] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:41:22,054] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:41:22,062] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.084067ms
[2022-09-10 15:41:22,088] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:41:22,114] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:41:22,176] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 59.653457ms
[2022-09-10 15:41:22,246] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:41:22,246] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.68444878566409, 'mem_free': 12863688704, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:41:22,246] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:41:22,250] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:22,251] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:22,856] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:22,861] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:22,861] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:23,703] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:23,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:23,711] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:24,794] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:24,804] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:24,805] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:25,898] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:32,444] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-10 15:41:32,590] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:41:33,494] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:41:33,523] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:41:33,523] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:41:33,583] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:41:33,640] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:41:33,696] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:41:33,697] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:41:33,794] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:41:33,795] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:41:33,822] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:41:33,960] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:41:33,961] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:41:33,988] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:41:34,015] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:41:34,015] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:41:34,043] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:41:34,070] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:41:34,070] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:41:34,098] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:41:34,126] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:41:34,126] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:41:34,155] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:41:34,155] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 15:41:34,155] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:41:34,156] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.617s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_9'}
[2022-09-10 15:41:34,243] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:34,243] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:34,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:34,945] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:41:35,028] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:41:35,028] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-10 15:41:35,029] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:41:35,269] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:41:35,298] - [task:165] INFO -  {'uptime': '756', 'memoryTotal': 15466930176, 'memoryFree': 12999475200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:41:35,323] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:41:35,324] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:41:35,324] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:41:35,362] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:41:35,397] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:41:35,398] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:41:35,426] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:41:35,427] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:41:35,427] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:41:35,427] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:41:35,475] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:41:35,478] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:35,478] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:36,079] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:36,080] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:41:36,201] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:41:36,202] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:41:36,232] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:41:36,232] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:41:36,258] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:41:36,344] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:41:36,375] - [task:165] INFO -  {'uptime': '752', 'memoryTotal': 15466930176, 'memoryFree': 12876709888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:41:36,401] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:41:36,435] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:41:36,435] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:41:36,486] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:41:36,489] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:36,489] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:37,109] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:37,110] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:41:37,235] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:41:37,236] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:41:37,267] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:41:37,267] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:41:37,294] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:41:37,378] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:41:37,404] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 12995239936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:41:37,430] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:41:37,457] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:41:37,457] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:41:37,509] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:41:37,514] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:37,514] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:38,128] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:38,129] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:41:38,245] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:41:38,247] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:41:38,275] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:41:38,276] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:41:38,302] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:41:38,385] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:41:38,411] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 12986523648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:41:38,436] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:41:38,463] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:41:38,463] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:41:38,518] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:41:38,521] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:38,521] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:39,159] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:39,160] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:41:39,286] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:41:39,288] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:41:39,317] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:41:39,318] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:41:39,343] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:41:39,399] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:41:39,895] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:41:44,901] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:41:44,983] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:41:44,987] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:44,988] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:45,607] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:45,608] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:41:45,726] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:41:45,727] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:41:45,727] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:41:46,538] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:41:46,592] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 15:41:46,593] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:41:47,476] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:41:47,775] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:41:48,033] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:41:48,036] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 15:41:48,092] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:48,092] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:48,747] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:48,752] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:48,752] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:49,636] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:49,645] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:49,645] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:50,757] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:50,767] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:41:50,767] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:41:51,867] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:41:58,418] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:41:58,418] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.65104670593278, 'mem_free': 13008211968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:41:58,418] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:41:58,418] - [basetestcase:467] INFO - Time to execute basesetup : 24.17853021621704
[2022-09-10 15:41:58,470] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:41:58,470] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:41:58,524] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:41:58,524] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:41:58,580] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:41:58,580] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:41:58,639] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:41:58,639] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:41:58,692] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:41:58,760] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:41:58,761] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:41:58,761] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:42:03,773] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:42:03,777] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:42:03,777] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:42:04,416] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:42:05,557] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:42:05,729] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:42:07,863] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:42:07,947] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:42:07,948] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:42:07,948] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:42:37,976] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:42:38,006] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:42:38,031] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:42:38,097] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.257289ms
[2022-09-10 15:42:38,097] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f64acdb7-b9ac-4a2f-a754-72e489d2a901', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.257289ms', 'executionTime': '64.195745ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:42:38,097] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:42:38,123] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:42:38,149] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:42:38,842] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 691.523538ms
[2022-09-10 15:42:38,842] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:42:38,907] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:42:38,943] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:42:38,952] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.502596ms
[2022-09-10 15:42:39,141] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:42:39,204] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:42:39,222] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:42:39,222] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:42:39,239] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:42:39,301] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:42:40,100] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee59bb31947d224372b0ecb3cc607fca3ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:42:40,125] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee59bb31947d224372b0ecb3cc607fca3ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:42:40,173] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.834749ms
[2022-09-10 15:42:40,173] - [base_gsi:282] INFO - BUILD INDEX on default(`employee59bb31947d224372b0ecb3cc607fca3ajoin_yr`) USING GSI
[2022-09-10 15:42:41,205] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee59bb31947d224372b0ecb3cc607fca3ajoin_yr`) USING GSI
[2022-09-10 15:42:41,232] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee59bb31947d224372b0ecb3cc607fca3ajoin_yr%60%29+USING+GSI
[2022-09-10 15:42:41,261] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.870766ms
[2022-09-10 15:42:41,311] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59bb31947d224372b0ecb3cc607fca3ajoin_yr'
[2022-09-10 15:42:41,341] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59bb31947d224372b0ecb3cc607fca3ajoin_yr%27
[2022-09-10 15:42:41,350] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.295437ms
[2022-09-10 15:42:42,381] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59bb31947d224372b0ecb3cc607fca3ajoin_yr'
[2022-09-10 15:42:42,410] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59bb31947d224372b0ecb3cc607fca3ajoin_yr%27
[2022-09-10 15:42:42,420] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.722759ms
[2022-09-10 15:42:43,451] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59bb31947d224372b0ecb3cc607fca3ajoin_yr'
[2022-09-10 15:42:43,477] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59bb31947d224372b0ecb3cc607fca3ajoin_yr%27
[2022-09-10 15:42:43,486] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.593451ms
[2022-09-10 15:42:44,518] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:42:44,544] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 15:42:44,548] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.805364ms
[2022-09-10 15:42:44,548] - [base_gsi:504] INFO - {'requestID': 'da8e312f-ff78-4fb3-9b24-fe3af53c21a4', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee59bb31947d224372b0ecb3cc607fca3ajoin_yr', 'index_id': '837993ef6be4dcda', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.805364ms', 'executionTime': '2.730135ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 15:42:44,549] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:42:44,550] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:42:44,550] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:42:44,550] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:42:44,551] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:42:44,552] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:42:44,552] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:42:44,552] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:42:44,610] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:42:44,650] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:42:44,677] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 15:42:44,855] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 165.056826ms
[2022-09-10 15:42:44,856] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:42:44,856] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:42:47,139] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:42:47,177] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59bb31947d224372b0ecb3cc607fca3ajoin_yr'
[2022-09-10 15:42:47,209] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59bb31947d224372b0ecb3cc607fca3ajoin_yr%27
[2022-09-10 15:42:47,214] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.151413ms
[2022-09-10 15:42:47,215] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7415a298-bb43-40fa-96d3-c2548287392f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.151413ms', 'executionTime': '4.083377ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:42:47,315] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:42:47,318] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:42:47,318] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:42:48,024] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:42:48,082] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:42:48,082] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:42:48,191] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:42:48,263] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:42:48,266] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:42:48,266] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:42:48,950] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:42:49,008] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:42:49,008] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:42:49,123] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:42:49,181] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:42:49,181] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:42:49,181] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:42:49,207] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:42:49,236] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:42:49,243] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.396324ms
[2022-09-10 15:42:49,243] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd1f97a62-cbb7-4f76-9124-427142a04314', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.396324ms', 'executionTime': '5.310577ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:42:49,298] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:42:49,299] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.117540242162429, 'mem_free': 12870995968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:42:49,299] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:42:49,305] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:42:49,305] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:42:49,995] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:42:50,000] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:42:50,000] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:42:50,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:42:50,805] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:42:50,805] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:42:51,955] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:42:51,966] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:42:51,966] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:42:53,112] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:00,315] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-10 15:43:00,449] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:43:00,501] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:43:00,559] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:43:00,614] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:43:00,614] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:43:00,693] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:43:00,694] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:43:00,721] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:43:00,851] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:43:00,852] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:43:00,879] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:43:00,904] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:43:00,904] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:43:00,930] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:43:00,957] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:43:00,958] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:43:00,986] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:43:01,011] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:43:01,012] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:43:01,038] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:43:01,038] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 15:43:01,039] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:43:01,039] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.850s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_10'}
[2022-09-10 15:43:01,117] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:01,118] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:01,800] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:01,834] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:43:01,915] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:43:01,916] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-10 15:43:01,916] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:43:02,149] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:43:02,177] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 12997525504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:43:02,203] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:43:02,203] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:43:02,204] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:43:02,242] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:43:02,277] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:43:02,277] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:43:02,306] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:43:02,307] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:43:02,307] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:43:02,308] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:43:02,357] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:43:02,360] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:02,361] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:02,986] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:02,988] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:43:03,126] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:43:03,127] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:43:03,159] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:43:03,159] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:43:03,186] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:43:03,275] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:43:03,302] - [task:165] INFO -  {'uptime': '842', 'memoryTotal': 15466930176, 'memoryFree': 12997173248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:43:03,330] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:43:03,360] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:43:03,360] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:43:03,413] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:43:03,418] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:03,418] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:04,105] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:04,106] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:43:04,236] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:43:04,237] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:43:04,267] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:43:04,268] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:43:04,297] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:43:04,387] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:43:04,414] - [task:165] INFO -  {'uptime': '839', 'memoryTotal': 15466930176, 'memoryFree': 12997910528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:43:04,440] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:43:04,470] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:43:04,470] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:43:04,521] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:43:04,525] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:04,525] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:05,161] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:05,162] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:43:05,280] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:43:05,281] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:43:05,310] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:43:05,311] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:43:05,337] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:43:05,420] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:43:05,446] - [task:165] INFO -  {'uptime': '838', 'memoryTotal': 15466930176, 'memoryFree': 12995772416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:43:05,472] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:43:05,500] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:43:05,501] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:43:05,553] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:43:05,555] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:05,556] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:06,192] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:06,193] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:43:06,320] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:43:06,321] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:43:06,350] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:43:06,351] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:43:06,376] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:43:06,432] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:43:06,970] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:43:11,974] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:43:12,056] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:43:12,060] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:12,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:12,727] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:12,728] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:43:12,850] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:43:12,851] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:43:12,851] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:43:13,550] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:43:13,607] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 15:43:13,607] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:43:14,540] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:43:14,867] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:43:15,167] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:43:15,171] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 15:43:15,234] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:15,235] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:15,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:15,915] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:15,916] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:16,932] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:16,943] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:16,943] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:18,075] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:18,083] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:18,083] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:19,233] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:26,148] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:43:26,149] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.22678264716937, 'mem_free': 12992032768, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:43:26,149] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:43:26,149] - [basetestcase:467] INFO - Time to execute basesetup : 25.034029960632324
[2022-09-10 15:43:26,202] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:43:26,202] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:43:26,257] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:43:26,257] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:43:26,308] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:43:26,309] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:43:26,360] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:43:26,361] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:43:26,412] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:43:26,471] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:43:26,472] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:43:26,472] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:43:31,479] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:43:31,483] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:43:31,483] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:43:32,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:43:33,245] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:43:33,540] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:43:36,601] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:43:36,677] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:43:36,677] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:43:36,677] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:44:06,707] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:44:06,736] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:44:06,761] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:44:06,828] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.676835ms
[2022-09-10 15:44:06,828] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b4f7042d-e0a3-4275-bcca-62e111017f6f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.676835ms', 'executionTime': '64.597737ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:44:06,828] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:44:06,854] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:44:06,879] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:44:07,558] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 676.610025ms
[2022-09-10 15:44:07,558] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:44:07,636] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:44:07,665] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:44:07,673] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.07149ms
[2022-09-10 15:44:07,881] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:44:07,918] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:44:07,942] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:44:07,943] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:44:07,961] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:44:08,024] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:44:08,842] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee2db5765d842a4fbc81b09df74c2c5c55join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:44:08,867] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee2db5765d842a4fbc81b09df74c2c5c55join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:44:08,931] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.214723ms
[2022-09-10 15:44:08,931] - [base_gsi:282] INFO - BUILD INDEX on default(`employee2db5765d842a4fbc81b09df74c2c5c55join_yr`) USING GSI
[2022-09-10 15:44:09,963] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee2db5765d842a4fbc81b09df74c2c5c55join_yr`) USING GSI
[2022-09-10 15:44:09,988] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee2db5765d842a4fbc81b09df74c2c5c55join_yr%60%29+USING+GSI
[2022-09-10 15:44:10,015] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.7632ms
[2022-09-10 15:44:10,059] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2db5765d842a4fbc81b09df74c2c5c55join_yr'
[2022-09-10 15:44:10,088] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2db5765d842a4fbc81b09df74c2c5c55join_yr%27
[2022-09-10 15:44:10,097] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.248445ms
[2022-09-10 15:44:11,129] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2db5765d842a4fbc81b09df74c2c5c55join_yr'
[2022-09-10 15:44:11,156] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2db5765d842a4fbc81b09df74c2c5c55join_yr%27
[2022-09-10 15:44:11,166] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.216178ms
[2022-09-10 15:44:12,196] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2db5765d842a4fbc81b09df74c2c5c55join_yr'
[2022-09-10 15:44:12,224] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2db5765d842a4fbc81b09df74c2c5c55join_yr%27
[2022-09-10 15:44:12,232] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.636381ms
[2022-09-10 15:44:13,263] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:44:13,290] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 15:44:13,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.498183ms
[2022-09-10 15:44:13,294] - [base_gsi:504] INFO - {'requestID': 'ea871d9c-c9e7-4522-9bfe-921bb06bb4b2', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2db5765d842a4fbc81b09df74c2c5c55join_yr', 'index_id': '21d7e1dd83633674', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.498183ms', 'executionTime': '2.421429ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 15:44:13,295] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:44:13,295] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:44:13,295] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:44:13,296] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:44:13,296] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:44:13,296] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:44:13,296] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:44:13,296] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:44:13,354] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:44:13,392] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:44:13,419] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 15:44:13,570] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 139.101307ms
[2022-09-10 15:44:13,571] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:44:13,571] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:44:14,774] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-10 15:44:14,830] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-10 15:44:14,863] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:44:14,892] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 15:44:14,895] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.354838ms
[2022-09-10 15:44:14,896] - [base_gsi:504] INFO - {'requestID': 'dbd92b13-368e-43ba-a954-43bd8d9d8829', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2db5765d842a4fbc81b09df74c2c5c55join_yr', 'index_id': '21d7e1dd83633674', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.354838ms', 'executionTime': '2.287746ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 15:44:14,896] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:44:14,896] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:44:14,896] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:44:14,897] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:44:14,897] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:44:14,897] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:44:14,897] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:44:14,897] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:44:14,950] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:44:14,988] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:44:15,015] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 15:44:15,132] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 108.30351ms
[2022-09-10 15:44:15,133] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:44:15,133] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:44:16,329] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee2db5765d842a4fbc81b09df74c2c5c55join_yr ON default USING GSI
[2022-09-10 15:44:16,356] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee2db5765d842a4fbc81b09df74c2c5c55join_yr+ON+default+USING+GSI
[2022-09-10 15:44:16,406] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.581939ms
[2022-09-10 15:44:16,441] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2db5765d842a4fbc81b09df74c2c5c55join_yr'
[2022-09-10 15:44:16,470] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2db5765d842a4fbc81b09df74c2c5c55join_yr%27
[2022-09-10 15:44:16,478] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.276298ms
[2022-09-10 15:44:16,479] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '43172424-cdce-4f93-b31e-e1fbb866eef9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.276298ms', 'executionTime': '6.200681ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:44:16,591] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:44:16,594] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:16,594] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:17,286] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:17,341] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:44:17,342] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:44:17,455] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:44:17,512] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:44:17,515] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:17,515] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:18,180] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:18,235] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:44:18,235] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:44:18,347] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:44:18,402] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:44:18,403] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:44:18,403] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:44:18,429] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:44:18,455] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:44:18,465] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.65865ms
[2022-09-10 15:44:18,495] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:44:18,520] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:44:18,571] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.197323ms
[2022-09-10 15:44:18,646] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:44:18,646] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.972666886463482, 'mem_free': 12832358400, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:44:18,646] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:44:18,650] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:18,650] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:19,304] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:19,309] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:19,309] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:20,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:20,463] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:20,463] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:21,651] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:21,659] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:21,659] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:22,877] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:29,728] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-10 15:44:29,869] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:44:31,152] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:44:31,180] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:44:31,180] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:44:31,241] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:44:31,296] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:44:31,348] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:44:31,349] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:44:31,428] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:44:31,429] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:44:31,454] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:44:31,584] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:44:31,584] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:44:31,610] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:44:31,635] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:44:31,635] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:44:31,662] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:44:31,688] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:44:31,688] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:44:31,714] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:44:31,739] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:44:31,739] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:44:31,765] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:44:31,765] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 15:44:31,766] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:44:31,766] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 90.702s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_11'}
[2022-09-10 15:44:31,844] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:31,844] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:32,555] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:32,588] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:44:32,666] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:44:32,667] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-10 15:44:32,667] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:44:32,870] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:44:32,896] - [task:165] INFO -  {'uptime': '933', 'memoryTotal': 15466930176, 'memoryFree': 12967596032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:44:32,921] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:44:32,921] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:44:32,922] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:44:32,955] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:44:32,990] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:44:32,991] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:44:33,020] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:44:33,021] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:44:33,021] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:44:33,022] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:44:33,069] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:44:33,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:33,072] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:33,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:33,721] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:44:33,846] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:44:33,847] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:44:33,877] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:44:33,877] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:44:33,903] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:44:33,990] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:44:34,017] - [task:165] INFO -  {'uptime': '932', 'memoryTotal': 15466930176, 'memoryFree': 12971659264, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:44:34,042] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:44:34,069] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:44:34,070] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:44:34,120] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:44:34,123] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:34,123] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:34,787] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:34,788] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:44:34,924] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:44:34,925] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:44:34,959] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:44:34,960] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:44:34,986] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:44:35,076] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:44:35,104] - [task:165] INFO -  {'uptime': '929', 'memoryTotal': 15466930176, 'memoryFree': 12972158976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:44:35,131] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:44:35,160] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:44:35,160] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:44:35,210] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:44:35,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:35,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:35,927] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:35,929] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:44:36,067] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:44:36,068] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:44:36,099] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:44:36,100] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:44:36,127] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:44:36,219] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:44:36,247] - [task:165] INFO -  {'uptime': '929', 'memoryTotal': 15466930176, 'memoryFree': 12972199936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:44:36,274] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:44:36,304] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:44:36,304] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:44:36,358] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:44:36,361] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:36,361] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:37,057] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:37,059] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:44:37,209] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:44:37,210] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:44:37,241] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:44:37,242] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:44:37,267] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:44:37,323] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:44:37,843] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:44:42,849] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:44:42,933] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:44:42,937] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:42,937] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:43,628] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:43,630] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:44:43,760] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:44:43,761] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:44:43,762] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:44:44,435] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:44:44,488] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 15:44:44,489] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:44:45,390] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:44:45,699] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:44:45,979] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:44:45,982] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 15:44:46,048] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:46,048] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:46,746] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:46,750] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:46,751] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:47,773] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:47,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:47,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:48,949] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:48,958] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:44:48,958] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:44:50,156] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:44:57,298] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:44:57,298] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.55331544668455, 'mem_free': 12967534592, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:44:57,298] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:44:57,299] - [basetestcase:467] INFO - Time to execute basesetup : 25.45677924156189
[2022-09-10 15:44:57,357] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:44:57,357] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:44:57,411] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:44:57,412] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:44:57,468] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:44:57,468] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:44:57,522] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:44:57,522] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:44:57,576] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:44:57,642] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:44:57,642] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:44:57,642] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:45:02,651] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:45:02,656] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:45:02,656] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:45:03,385] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:45:04,559] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:45:04,733] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:45:07,723] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:45:07,804] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:07,805] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:45:07,805] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:45:37,824] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:45:37,854] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:45:37,881] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:45:37,950] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.478289ms
[2022-09-10 15:45:37,951] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '53cd4e20-e1fc-448b-8ebd-1cd8224fcfe5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.478289ms', 'executionTime': '67.418358ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:45:37,951] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:45:37,977] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:45:38,002] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:45:38,715] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 711.157444ms
[2022-09-10 15:45:38,716] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:45:38,773] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:45:38,813] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:45:38,823] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.084528ms
[2022-09-10 15:45:39,023] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:39,062] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:45:39,083] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:45:39,083] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:45:39,099] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:45:39,168] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:39,984] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 15:45:40,009] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 15:45:40,078] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.854233ms
[2022-09-10 15:45:40,079] - [base_gsi:282] INFO - BUILD INDEX on default(`employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr`) USING GSI
[2022-09-10 15:45:41,109] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr`) USING GSI
[2022-09-10 15:45:41,134] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr%60%29+USING+GSI
[2022-09-10 15:45:41,158] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.403698ms
[2022-09-10 15:45:41,198] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr'
[2022-09-10 15:45:41,233] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr%27
[2022-09-10 15:45:41,243] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.881046ms
[2022-09-10 15:45:42,273] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr'
[2022-09-10 15:45:42,298] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr%27
[2022-09-10 15:45:42,305] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.381732ms
[2022-09-10 15:45:43,335] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr'
[2022-09-10 15:45:43,360] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr%27
[2022-09-10 15:45:43,367] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.361515ms
[2022-09-10 15:45:44,396] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:45:44,421] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 15:45:44,426] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.730895ms
[2022-09-10 15:45:44,426] - [base_gsi:504] INFO - {'requestID': '38bace73-4886-42e7-8aa0-fa063f6ad9e5', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr', 'index_id': '88b63cb66ac0749d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.730895ms', 'executionTime': '2.618534ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 15:45:44,426] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 15:45:44,427] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:45:44,427] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 15:45:44,427] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 15:45:44,427] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:45:44,428] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:45:44,428] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:45:44,428] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 15:45:44,489] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 15:45:44,527] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 15:45:44,554] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 15:45:44,723] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 155.785175ms
[2022-09-10 15:45:44,723] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 15:45:44,724] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 15:45:46,988] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 15:45:49,487] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:45:49,548] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 15:45:49,558] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:45:50,557] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:45:50,871] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:45:51,137] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:45:51,140] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 15:45:53,173] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:45:53,173] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:45:53,201] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:45:53,228] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:45:53,228] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:45:53,254] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:45:53,282] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:45:53,282] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:45:53,308] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:45:53,335] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:45:53,335] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:45:53,361] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:45:53,412] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:53,436] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr'
[2022-09-10 15:45:53,461] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr%27
[2022-09-10 15:45:53,504] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.004218ms
[2022-09-10 15:45:53,504] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '88ae88df-8627-44de-9d07-1efd6c314b12', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '42.004218ms', 'executionTime': '41.946047ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:45:53,532] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr'
[2022-09-10 15:45:53,558] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb994cb9b691247b0a8e5e957ac65169fjoin_yr%27
[2022-09-10 15:45:53,560] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.176842ms
[2022-09-10 15:45:53,561] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0ed45ede-23b5-47e3-b1de-331d030211ac', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.176842ms', 'executionTime': '1.109791ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:45:53,612] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:53,742] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:53,744] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:45:53,744] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:45:54,445] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:45:54,504] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:45:54,504] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:45:54,620] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:45:54,677] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:54,680] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:45:54,681] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:45:55,366] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:45:55,421] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:45:55,421] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:45:55,537] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:45:55,594] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:45:55,594] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:45:55,595] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:45:55,620] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:45:55,645] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:45:55,654] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.952688ms
[2022-09-10 15:45:55,654] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b48a70ec-7e92-42bd-a84c-e64c1dfc989c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.952688ms', 'executionTime': '6.883944ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:45:55,654] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:45:55,680] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:45:55,705] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:45:55,707] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 999.499µs
[2022-09-10 15:45:55,707] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd6f31891-a71a-4ea7-937f-e0ba23693e91', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '999.499µs', 'executionTime': '954.626µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:45:55,756] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:45:55,756] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.48235944795941, 'mem_free': 12802428928, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:45:55,757] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:45:55,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:45:55,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:45:56,452] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:45:56,457] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:45:56,457] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:45:57,173] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:45:57,177] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:45:57,178] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:45:58,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:45:58,463] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:45:58,463] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:45:59,715] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:07,580] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 15:46:07,725] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:46:08,027] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:46:08,055] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:46:08,055] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:46:08,109] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:46:08,161] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:46:08,214] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:46:08,215] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:46:08,292] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:46:08,293] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:46:08,318] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:46:08,447] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:46:08,447] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:46:08,474] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:46:08,499] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:46:08,499] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:46:08,526] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:46:08,552] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:46:08,552] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:46:08,580] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:46:08,606] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:46:08,606] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:46:08,634] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:46:08,634] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 15:46:08,634] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:46:08,635] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.847s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_15-28-54/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-10 15:46:08,695] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:08,695] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:09,402] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:09,434] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:46:09,511] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 15:46:09,511] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-10 15:46:09,512] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 15:46:09,721] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 15:46:09,749] - [task:165] INFO -  {'uptime': '1030', 'memoryTotal': 15466930176, 'memoryFree': 12960141312, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:46:09,774] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 15:46:09,774] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 15:46:09,775] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 15:46:09,805] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 15:46:09,840] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 15:46:09,841] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 15:46:09,869] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 15:46:09,870] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 15:46:09,870] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 15:46:09,870] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 15:46:09,918] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:46:09,921] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:09,921] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:10,620] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:10,621] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:46:10,760] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:46:10,761] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:46:10,793] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:46:10,793] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:46:10,819] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:46:10,906] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 15:46:10,933] - [task:165] INFO -  {'uptime': '1028', 'memoryTotal': 15466930176, 'memoryFree': 12960546816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:46:10,959] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:46:10,989] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 15:46:10,990] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 15:46:11,042] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:46:11,045] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:11,045] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:11,792] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:11,793] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:46:11,932] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:46:11,933] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:46:11,962] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:46:11,963] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:46:11,990] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:46:12,076] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 15:46:12,104] - [task:165] INFO -  {'uptime': '1030', 'memoryTotal': 15466930176, 'memoryFree': 12960169984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:46:12,132] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:46:12,162] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 15:46:12,163] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 15:46:12,215] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:46:12,218] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:12,218] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:12,972] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:12,973] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:46:13,111] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:46:13,112] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:46:13,142] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:46:13,143] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:46:13,168] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:46:13,260] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 15:46:13,288] - [task:165] INFO -  {'uptime': '1029', 'memoryTotal': 15466930176, 'memoryFree': 12960219136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 15:46:13,313] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 15:46:13,341] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 15:46:13,341] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 15:46:13,391] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 15:46:13,394] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:13,394] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:14,111] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:14,113] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:46:14,247] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:46:14,249] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:46:14,279] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 15:46:14,279] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 15:46:14,305] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 15:46:14,363] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 15:46:14,877] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:46:19,881] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 15:46:19,965] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 15:46:19,970] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:19,970] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:20,680] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:20,681] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 15:46:20,817] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 15:46:20,819] - [remote_util:5249] INFO - b'ok'
[2022-09-10 15:46:20,819] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 15:46:21,485] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 15:46:21,549] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 15:46:21,550] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 15:46:22,463] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:46:22,790] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:46:23,080] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 15:46:23,085] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-10 15:46:23,148] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:23,148] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:23,866] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:23,870] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:23,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:25,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:25,097] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:25,097] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:26,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:26,389] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:26,389] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:27,658] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:34,971] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:46:34,972] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.78395171833751, 'mem_free': 12959772672, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:46:34,972] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:46:34,972] - [basetestcase:467] INFO - Time to execute basesetup : 26.279984951019287
[2022-09-10 15:46:35,022] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:46:35,022] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:46:35,075] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:46:35,075] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:46:35,129] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:46:35,129] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:46:35,182] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 15:46:35,182] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 15:46:35,233] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:46:35,289] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 15:46:35,289] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 15:46:35,290] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 15:46:40,301] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 15:46:40,305] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:46:40,305] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:46:41,031] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:46:42,219] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 15:46:42,388] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 15:46:45,355] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 15:46:45,431] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:46:45,431] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 15:46:45,431] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 15:47:15,461] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 15:47:15,491] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:47:15,517] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:47:15,585] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.001107ms
[2022-09-10 15:47:15,585] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '76753a28-d5c0-490b-ab78-1568893830e0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.001107ms', 'executionTime': '65.941847ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 15:47:15,585] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 15:47:15,612] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 15:47:15,637] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 15:47:16,373] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 733.593959ms
[2022-09-10 15:47:16,374] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 15:47:16,463] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:47:16,499] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:47:16,507] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.024515ms
[2022-09-10 15:47:16,725] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:47:16,760] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 15:47:16,776] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 15:47:16,776] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 15:47:16,790] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 15:47:16,869] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:47:17,000] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:47:17,003] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:47:17,003] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:47:17,830] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:47:17,887] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:47:17,887] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 15:47:18,016] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:47:18,070] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:47:18,073] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:47:18,073] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:47:18,867] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:47:18,925] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 15:47:18,925] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 15:47:19,056] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 15:47:19,110] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 15:47:19,111] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 15:47:19,111] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:47:19,135] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 15:47:19,162] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 15:47:19,173] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.717955ms
[2022-09-10 15:47:19,199] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 15:47:19,224] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 15:47:19,273] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.14124ms
[2022-09-10 15:47:19,334] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 15:47:19,334] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.88327422562063, 'mem_free': 12760215552, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 15:47:19,334] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 15:47:19,338] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:47:19,339] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:47:20,107] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:47:20,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:47:20,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:47:20,880] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:47:20,885] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:47:20,885] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:47:22,080] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:47:22,092] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 15:47:22,092] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 15:47:23,452] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 15:47:32,137] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-10 15:47:32,327] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 15:47:33,034] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 15:47:33,063] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 15:47:33,063] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 15:47:33,122] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:47:33,181] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:47:33,238] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 15:47:33,239] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 15:47:33,322] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 15:47:33,324] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 15:47:33,350] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:47:33,501] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 15:47:33,501] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:47:33,528] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 15:47:33,556] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 15:47:33,556] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:47:33,584] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 15:47:33,612] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 15:47:33,612] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:47:33,640] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 15:47:33,668] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 15:47:33,668] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 15:47:33,695] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 15:47:33,695] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-10 15:47:33,696] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 15:47:33,696] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.080s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 19228 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-10.09.2022-13.58.cfg
Build Log: make-10.09.2022-13.58.log
Server Log: logs-10.09.2022-13.58.tar.gz

Finished