Building

Started building at 2022/09/10 16:00:09
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 10.09.2022-16.00
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at de1130b, changes since last good build: none
Switching plasma to unstable
plasma is at a338d0e, changes since last good build: 
fatal: Invalid revision range 2455f58ea891433394e537dcd7d420fc1e15b74e..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable 7c7f0f91] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 15d12608] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 26ddf575] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable c4b0b70e] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable 1f5949d6] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 83b73fd0] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable b50bde7c] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable e49c158b] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable b1e9882a] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 0143bbed] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable 658777fb] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable e146ecd0] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179227 at refs/changes/27/179227/10
Total 8 (delta 3), reused 4 (delta 3)
[unstable 6885ea97] MB-51825: Remove vbuckets arg from indexer command line and its usage
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Aug 25 18:17:32 2022 +0530
 2 files changed, 9 deletions(-)
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable df2b7284] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 97553ed] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable cc3c002] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable b6df1c6] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 4bb6f0c] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable be0f628] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable 61eb259] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable 4ea98ab] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable ec01251] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Pulling in uncommitted change 179769 at refs/changes/69/179769/2
Total 5 (delta 4), reused 5 (delta 4)
[unstable 70ba222] MB-52096: Use memory stats to estimate mutation quota if page stats not available
 Author: jliang00 
 Date: Thu Sep 8 16:56:16 2022 -0700
 3 files changed, 86 insertions(+), 27 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/10 16:54:56
Testing mode: sanity,unit,functional,integration
Using storage type: plasma
Setting ulimit to 200000

Simple Test

Sep 10 17:00:21 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 17:04:14 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 17:04:58 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 10 17:06:27 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 10 17:07:15 b" 'stop_on_failure': 'True'}"
Sep 10 17:07:15 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 17:07:15 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_1'}"
Sep 10 17:07:15 b'-->result: '
Sep 10 17:07:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 10 17:07:15 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 17:07:15 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_2'}"
Sep 10 17:07:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 17:07:15 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 10 17:07:15 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_3'}"
Sep 10 17:07:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 17:07:15 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 17:07:15 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 10 17:07:15 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_4'}"
Sep 10 17:07:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 17:07:15 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 17:07:15 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 17:17:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 10 17:17:48 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 10 17:21:34 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 17:25:29 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 17:28:12 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 17:28:12 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_5'}"
Sep 10 17:28:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 17:28:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 10 17:28:12 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 10 17:28:12 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_6'}"
Sep 10 17:28:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 17:28:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 17:28:12 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 10 17:28:12 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_7'}"
Sep 10 17:28:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 17:28:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 17:28:12 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 10 17:28:12 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 10 17:28:12 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_16-55-18/test_8'}"
Sep 10 17:28:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 17:28:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 17:28:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 17:28:12 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 10 17:28:12 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 10 17:28:13 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 17:28:13 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 17:28:13 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 10 17:28:13 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 10 17:28:13 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 10 17:28:13 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 10 17:28:13 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 10 17:28:13 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 187: go: command not found
/home/buildbot/bin/dotest: line 190: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=plasma
Printing gsi_type=plasma
gsi_type=plasma
In here
-p makefile=True,gsi_type=plasma
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'plasma',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-10 17:31:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:04 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 17:31:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 17:31:13 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:25 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 17:31:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:33 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-10 17:31:34 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 2159669248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:31:34 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:31:35 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 13503102976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:31:35 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 17:31:35 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:31:36 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '21', 'memoryTotal': 15466930176, 'memoryFree': 1854726144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:31:36 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:31:37 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1767038976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:31:37 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:31:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:31:38 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:31:38 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:38 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:31:38 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:31:38 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 17:31:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 17:31:38 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-10 17:31:38 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 17:31:43 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 17:31:43 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-10 17:31:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:44 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:31:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:31:44 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:31:44 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 17:31:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 17:31:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-10 17:31:44 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 17:31:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 17:31:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 17:31:45 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 17:31:45 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 0, 'mem_free': 13391679488, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.86086988449097
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 17:31:49 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 17:31:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 17:31:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:31:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:31:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:31:55 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 17:31:55 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 17:31:57 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 17:31:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:31:57 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 17:31:57 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 17:32:27 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 17:32:27 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 17:32:27 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 140.264471ms
2022-09-10 17:32:28 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'bdcac6fd-58df-48f4-a35d-fc7d54549304', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '140.264471ms', 'executionTime': '140.063501ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 743.558262ms
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 17:32:28 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.883155ms
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.484852ms
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 48.011547ms
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.915815655642972, 'mem_free': 13366104064, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:34 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 17:32:34 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 17:32:36 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 17:32:36 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 92.181s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-10 17:32:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:36 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.972081244899115, 'mem_free': 13360812032, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:41 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 17:32:42 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 17:32:42 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13335789568, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 17:32:43 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13337460736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13342461952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13343621120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:32:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:32:46 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:32:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 17:32:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 17:32:46 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 17:32:46 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 17:32:51 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 17:32:51 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-10 17:32:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 17:32:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 17:32:52 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 17:32:52 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 17:32:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 17:32:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-10 17:32:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 17:32:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 17:32:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 17:32:53 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 17:32:53 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 17:32:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:32:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.95435462807436, 'mem_free': 13318488064, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.557472705841064
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 17:32:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 17:33:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 17:33:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:33:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:33:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:33:04 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 17:33:05 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 17:33:07 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 17:33:07 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:33:07 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 17:33:07 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 17:33:37 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 17:33:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 17:33:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 17:33:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 66.80415ms
2022-09-10 17:33:37 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '5261ce48-f638-47df-be58-b34a051791d1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.80415ms', 'executionTime': '66.656824ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 17:33:37 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 17:33:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 17:33:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 688.921218ms
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.685316ms
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 17:33:38 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 17:33:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:33:39 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee1e4c3730b7e146c492f5927e7e8e8831job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-10 17:33:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee1e4c3730b7e146c492f5927e7e8e8831job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-10 17:33:39 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 59.802196ms
2022-09-10 17:33:39 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee1e4c3730b7e146c492f5927e7e8e8831job_title) USING GSI
2022-09-10 17:33:40 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee1e4c3730b7e146c492f5927e7e8e8831job_title) USING GSI
2022-09-10 17:33:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee1e4c3730b7e146c492f5927e7e8e8831job_title%29+USING+GSI
2022-09-10 17:33:41 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 24.666267ms
2022-09-10 17:33:42 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee1e4c3730b7e146c492f5927e7e8e8831job_title'
2022-09-10 17:33:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1e4c3730b7e146c492f5927e7e8e8831job_title%27
2022-09-10 17:33:42 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.957213ms
2022-09-10 17:33:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee1e4c3730b7e146c492f5927e7e8e8831job_title'
2022-09-10 17:33:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1e4c3730b7e146c492f5927e7e8e8831job_title%27
2022-09-10 17:33:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.471415ms
2022-09-10 17:33:44 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 17:33:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 17:33:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-10 17:33:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.498882ms
2022-09-10 17:33:44 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': 'dda41cb9-a8f6-40eb-8083-482e1e3434a7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee1e4c3730b7e146c492f5927e7e8e8831job_title', 'index_id': '393788301fa7c7c6', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.498882ms', 'executionTime': '2.351518ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-10 17:33:44 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 17:33:44 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 17:33:44 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 17:33:44 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-10 17:33:44 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-10 17:33:44 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-10 17:33:44 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-10 17:33:44 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-10 17:33:44 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-10 17:33:45 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 17:33:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 17:33:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-10 17:33:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 161.250003ms
2022-09-10 17:33:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-10 17:33:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-10 17:33:46 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 17:33:46 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee1e4c3730b7e146c492f5927e7e8e8831job_title'
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1e4c3730b7e146c492f5927e7e8e8831job_title%27
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.061176ms
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee1e4c3730b7e146c492f5927e7e8e8831job_title ON default USING GSI
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee1e4c3730b7e146c492f5927e7e8e8831job_title+ON+default+USING+GSI
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 31.081325ms
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee1e4c3730b7e146c492f5927e7e8e8831job_title'
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1e4c3730b7e146c492f5927e7e8e8831job_title%27
2022-09-10 17:33:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.879053ms
2022-09-10 17:33:47 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'd121212d-0ccb-4589-a637-6ebb0b916a01', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.879053ms', 'executionTime': '5.804156ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 17:33:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:33:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:33:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:33:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:33:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 17:33:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 17:33:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.512091ms
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 54.566085ms
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.72949477299997, 'mem_free': 13196996608, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:33:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 17:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 17:33:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 17:33:53 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 17:33:53 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 17:33:55 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 17:33:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 78.980s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_2'}
[2022-09-10 17:33:55,715] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:33:55,715] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:33:55,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:33:56,018] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:33:56,100] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:33:56,101] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-10 17:33:56,101] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:33:56,746] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:33:56,774] - [task:165] INFO -  {'uptime': '168', 'memoryTotal': 15466930176, 'memoryFree': 13196996608, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:33:56,800] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:33:56,800] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:33:56,801] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:33:56,832] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:33:56,866] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:33:56,867] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:33:56,897] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:33:56,898] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:33:56,898] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:33:56,898] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:33:56,950] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:33:56,953] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:33:56,953] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:33:57,243] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:33:57,244] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:33:57,321] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:33:57,322] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:33:57,353] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:57,381] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:57,411] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:33:57,547] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:33:57,575] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13249941504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:33:57,604] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:33:57,633] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:33:57,633] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:33:57,686] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:33:57,689] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:33:57,689] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:33:57,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:33:57,986] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:33:58,057] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:33:58,058] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:33:58,090] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:58,118] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:58,147] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:33:58,273] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:33:58,303] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13250625536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:33:58,332] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:33:58,361] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:33:58,362] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:33:58,415] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:33:58,418] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:33:58,418] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:33:58,700] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:33:58,701] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:33:58,775] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:33:58,776] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:33:58,806] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:58,832] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:58,860] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:33:58,976] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:33:59,003] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13246869504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:33:59,029] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:33:59,057] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:33:59,060] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:33:59,112] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:33:59,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:33:59,115] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:33:59,383] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:33:59,385] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:33:59,456] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:33:59,457] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:33:59,488] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:59,514] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:33:59,544] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:33:59,641] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:34:00,183] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:34:05,186] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:34:05,273] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:34:05,279] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:34:05,279] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:34:05,571] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:34:05,572] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:34:05,646] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:34:05,647] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:34:05,648] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:34:06,756] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:34:06,811] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 17:34:06,811] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:34:07,813] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:34:08,133] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:34:08,422] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:34:08,425] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:34:08,481] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:34:08,482] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:34:08,745] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:34:08,750] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:34:08,751] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:34:09,029] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:34:09,034] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:34:09,034] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:34:09,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:34:09,329] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:34:09,329] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:34:09,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:34:13,614] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:34:13,615] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.25206426710593, 'mem_free': 13250789376, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:34:13,615] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:34:13,615] - [basetestcase:467] INFO - Time to execute basesetup : 17.902281761169434
[2022-09-10 17:34:13,670] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:34:13,670] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:34:13,725] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:34:13,725] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:34:13,795] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:34:13,795] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:34:13,849] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:34:13,849] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:34:13,900] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:34:13,974] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:34:13,974] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:34:13,974] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:34:18,987] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:34:18,990] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:34:18,990] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:34:19,239] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:34:20,177] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:34:20,342] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:34:22,975] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:34:22,998] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 17:34:23,644] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:34:23,644] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:34:23,645] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:34:53,673] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:34:53,704] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:34:53,729] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:34:53,798] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.604066ms
[2022-09-10 17:34:53,798] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3fc4d638-691f-432e-8c4a-dba9a3c4ec9b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.604066ms', 'executionTime': '66.519844ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:34:53,798] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:34:53,824] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:34:53,849] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:34:54,597] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 745.632689ms
[2022-09-10 17:34:54,597] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:34:54,650] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:34:54,689] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:34:54,698] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.093716ms
[2022-09-10 17:34:54,922] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:34:54,976] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:34:54,994] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:34:54,995] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:34:55,017] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:34:55,094] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:34:55,873] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb0a23daa67bc404e94a87400c72c4507job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:34:55,898] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb0a23daa67bc404e94a87400c72c4507job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:34:55,950] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.248476ms
[2022-09-10 17:34:55,950] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb0a23daa67bc404e94a87400c72c4507job_title) USING GSI
[2022-09-10 17:34:56,981] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb0a23daa67bc404e94a87400c72c4507job_title) USING GSI
[2022-09-10 17:34:57,007] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb0a23daa67bc404e94a87400c72c4507job_title%29+USING+GSI
[2022-09-10 17:34:57,030] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.146862ms
[2022-09-10 17:34:58,062] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb0a23daa67bc404e94a87400c72c4507job_title'
[2022-09-10 17:34:58,089] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb0a23daa67bc404e94a87400c72c4507job_title%27
[2022-09-10 17:34:58,100] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.710938ms
[2022-09-10 17:34:59,130] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb0a23daa67bc404e94a87400c72c4507job_title'
[2022-09-10 17:34:59,157] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb0a23daa67bc404e94a87400c72c4507job_title%27
[2022-09-10 17:34:59,167] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.324227ms
[2022-09-10 17:34:59,609] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 17:34:59,774] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:35:00,659] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:35:01,169] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:35:01,205] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:35:01,234] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 17:35:01,240] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.766732ms
[2022-09-10 17:35:01,240] - [task:3246] INFO - {'requestID': '76cf20ce-e562-496c-ba07-9be2172b68e7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb0a23daa67bc404e94a87400c72c4507job_title', 'index_id': 'd0b350889461b20c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.766732ms', 'executionTime': '3.700323ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 17:35:01,240] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:35:01,240] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:35:01,241] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:35:01,241] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:35:01,242] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 17:35:01,242] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:35:01,242] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:35:01,242] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:35:01,243] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:35:02,242] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:35:02,273] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:35:02,301] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 17:35:02,464] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 152.859118ms
[2022-09-10 17:35:02,465] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:35:02,465] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:35:03,276] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:35:03,277] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:35:04,308] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb0a23daa67bc404e94a87400c72c4507job_title'
[2022-09-10 17:35:04,334] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb0a23daa67bc404e94a87400c72c4507job_title%27
[2022-09-10 17:35:04,344] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.405116ms
[2022-09-10 17:35:04,370] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb0a23daa67bc404e94a87400c72c4507job_title ON default USING GSI
[2022-09-10 17:35:04,395] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeb0a23daa67bc404e94a87400c72c4507job_title+ON+default+USING+GSI
[2022-09-10 17:35:04,436] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.087972ms
[2022-09-10 17:35:04,469] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb0a23daa67bc404e94a87400c72c4507job_title'
[2022-09-10 17:35:04,496] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb0a23daa67bc404e94a87400c72c4507job_title%27
[2022-09-10 17:35:04,504] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.692486ms
[2022-09-10 17:35:04,504] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8c8ea2c7-a154-4864-b835-ed3cbf6a47be', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.692486ms', 'executionTime': '6.617609ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:35:04,610] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:35:04,613] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:04,614] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:04,984] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:05,046] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:35:05,046] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:35:05,109] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:35:05,165] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:35:05,168] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:05,168] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:05,532] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:05,592] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:35:05,593] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:35:05,658] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:35:05,716] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:35:05,717] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:35:05,717] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:35:05,745] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:35:05,772] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:35:05,780] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.598492ms
[2022-09-10 17:35:05,807] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:35:05,832] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:35:05,881] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.683106ms
[2022-09-10 17:35:05,951] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:35:05,951] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.55465639470754, 'mem_free': 13111873536, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:35:05,952] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:35:05,956] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:05,956] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:06,295] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:06,300] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:06,300] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:06,627] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:06,631] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:06,631] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:07,209] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:07,219] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:07,220] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:07,823] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:11,768] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:35:11,910] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:35:12,467] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:35:12,497] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:35:12,497] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:35:12,552] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:35:12,604] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:35:12,661] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:35:12,662] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:35:12,742] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:35:12,743] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:35:12,771] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:35:12,900] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:35:12,900] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:35:12,927] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:35:12,952] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:35:12,953] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:35:12,979] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:35:13,007] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:35:13,007] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:35:13,044] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:35:13,072] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:35:13,072] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:35:13,098] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:35:13,099] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:35:13,099] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:35:13,099] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 77.437s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_3'}
[2022-09-10 17:35:13,197] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:13,197] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:13,503] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:13,534] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:35:13,614] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:35:13,615] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-10 17:35:13,615] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:35:14,228] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:35:14,257] - [task:165] INFO -  {'uptime': '245', 'memoryTotal': 15466930176, 'memoryFree': 13186486272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:35:14,283] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:35:14,284] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:35:14,284] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:35:14,321] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:35:14,357] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:35:14,358] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:35:14,385] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:35:14,386] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:35:14,386] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:35:14,386] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:35:14,435] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:35:14,438] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:14,438] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:14,752] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:14,754] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:35:14,829] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:35:14,830] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:35:14,860] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:14,887] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:14,918] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:35:15,048] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:35:15,080] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13123473408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:35:15,106] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:35:15,136] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:35:15,136] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:35:15,187] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:35:15,190] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:15,190] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:15,494] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:15,495] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:35:15,568] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:35:15,570] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:35:15,601] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:15,627] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:15,656] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:35:15,772] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:35:15,798] - [task:165] INFO -  {'uptime': '242', 'memoryTotal': 15466930176, 'memoryFree': 13123727360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:35:15,823] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:35:15,850] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:35:15,851] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:35:15,902] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:35:15,907] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:15,907] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:16,213] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:16,214] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:35:16,292] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:35:16,293] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:35:16,322] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:16,349] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:16,376] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:35:16,490] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:35:16,517] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13193646080, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:35:16,542] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:35:16,572] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:35:16,572] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:35:16,622] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:35:16,625] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:16,625] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:16,929] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:16,930] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:35:17,006] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:35:17,007] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:35:17,039] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:17,065] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:35:17,094] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:35:17,190] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:35:17,791] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:35:22,793] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:35:22,881] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:35:22,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:22,887] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:23,250] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:23,251] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:35:23,333] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:35:23,334] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:35:23,334] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:35:24,307] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:35:24,362] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 17:35:24,362] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:35:25,278] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:35:25,598] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:35:25,872] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:35:25,875] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:35:25,931] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:25,931] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:26,262] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:26,268] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:26,268] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:26,586] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:26,591] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:26,591] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:26,936] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:26,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:26,945] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:27,482] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:31,423] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:35:31,423] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.7483142466661, 'mem_free': 13185777664, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:35:31,423] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:35:31,424] - [basetestcase:467] INFO - Time to execute basesetup : 18.229125261306763
[2022-09-10 17:35:31,475] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:35:31,475] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:35:31,530] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:35:31,531] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:35:31,583] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:35:31,583] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:35:31,636] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:35:31,636] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:35:31,688] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:35:31,747] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:35:31,747] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:35:31,747] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:35:36,759] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:35:36,762] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:35:36,762] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:35:37,066] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:35:38,021] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:35:38,197] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:35:40,801] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:35:40,821] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 17:35:41,660] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:35:41,660] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:35:41,660] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:36:11,691] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:36:11,720] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:36:11,746] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:36:11,814] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.857574ms
[2022-09-10 17:36:11,815] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f4ae9546-88be-49fd-a003-62c5b4b2da91', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.857574ms', 'executionTime': '66.799038ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:36:11,815] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:36:11,841] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:36:11,866] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:36:12,559] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 691.055695ms
[2022-09-10 17:36:12,559] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:36:12,609] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:36:12,651] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:36:12,661] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.807547ms
[2022-09-10 17:36:12,895] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:36:12,933] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:36:12,958] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:36:12,959] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:36:12,974] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:36:13,036] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:36:13,851] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee20ed1006f4044e9799fb930a8a2a5c37job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:36:13,876] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee20ed1006f4044e9799fb930a8a2a5c37job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:36:13,948] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.49357ms
[2022-09-10 17:36:13,949] - [base_gsi:282] INFO - BUILD INDEX on default(employee20ed1006f4044e9799fb930a8a2a5c37job_title) USING GSI
[2022-09-10 17:36:14,981] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee20ed1006f4044e9799fb930a8a2a5c37job_title) USING GSI
[2022-09-10 17:36:15,008] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee20ed1006f4044e9799fb930a8a2a5c37job_title%29+USING+GSI
[2022-09-10 17:36:15,027] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.277979ms
[2022-09-10 17:36:16,058] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee20ed1006f4044e9799fb930a8a2a5c37job_title'
[2022-09-10 17:36:16,083] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee20ed1006f4044e9799fb930a8a2a5c37job_title%27
[2022-09-10 17:36:16,093] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.322284ms
[2022-09-10 17:36:17,123] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee20ed1006f4044e9799fb930a8a2a5c37job_title'
[2022-09-10 17:36:17,154] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee20ed1006f4044e9799fb930a8a2a5c37job_title%27
[2022-09-10 17:36:17,164] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.514042ms
[2022-09-10 17:36:17,532] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 17:36:17,784] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:36:18,484] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:36:19,167] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:36:19,201] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:36:19,228] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 17:36:19,232] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.209361ms
[2022-09-10 17:36:19,232] - [task:3246] INFO - {'requestID': 'eead0570-c00a-47eb-b3d8-f9e6749aa41e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee20ed1006f4044e9799fb930a8a2a5c37job_title', 'index_id': 'e18c41522ada1477', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.209361ms', 'executionTime': '2.14503ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 17:36:19,232] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:36:19,233] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:36:19,233] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:36:19,233] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:36:19,234] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 17:36:19,234] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:36:19,234] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:36:19,234] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:36:19,235] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:36:20,234] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:36:20,264] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:36:20,291] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 17:36:20,406] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 104.348214ms
[2022-09-10 17:36:20,406] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:36:20,407] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:36:21,167] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:36:21,168] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:36:22,200] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee20ed1006f4044e9799fb930a8a2a5c37job_title'
[2022-09-10 17:36:22,226] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee20ed1006f4044e9799fb930a8a2a5c37job_title%27
[2022-09-10 17:36:22,236] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.728934ms
[2022-09-10 17:36:22,261] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee20ed1006f4044e9799fb930a8a2a5c37job_title ON default USING GSI
[2022-09-10 17:36:22,286] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee20ed1006f4044e9799fb930a8a2a5c37job_title+ON+default+USING+GSI
[2022-09-10 17:36:22,347] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 59.488635ms
[2022-09-10 17:36:22,380] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee20ed1006f4044e9799fb930a8a2a5c37job_title'
[2022-09-10 17:36:22,410] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee20ed1006f4044e9799fb930a8a2a5c37job_title%27
[2022-09-10 17:36:22,417] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.785475ms
[2022-09-10 17:36:22,418] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '66341846-99a7-4a7e-93e2-df38809a47dc', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.785475ms', 'executionTime': '5.664012ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:36:22,521] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:36:22,524] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:22,525] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:22,930] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:22,986] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:36:22,986] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:36:23,065] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:36:23,121] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:36:23,124] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:23,124] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:23,518] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:23,576] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:36:23,577] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:36:23,645] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:36:23,701] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:36:23,701] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:36:23,701] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:36:23,727] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:36:23,753] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:36:23,762] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.652647ms
[2022-09-10 17:36:23,788] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:36:23,813] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:36:23,859] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.968691ms
[2022-09-10 17:36:23,926] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:36:23,926] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.20336316442116, 'mem_free': 13063696384, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:36:23,927] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:36:23,931] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:23,931] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:24,320] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:24,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:24,325] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:24,877] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:24,885] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:24,885] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:25,592] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:25,600] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:25,600] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:26,311] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:30,635] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:36:30,776] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:36:31,952] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:36:31,979] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:36:31,980] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:36:32,039] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:36:32,092] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:36:32,146] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:36:32,147] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:36:32,226] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:36:32,227] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:36:32,252] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:36:32,380] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:36:32,381] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:36:32,407] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:36:32,432] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:36:32,433] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:36:32,459] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:36:32,485] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:36:32,486] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:36:32,513] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:36:32,539] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:36:32,540] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:36:32,567] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:36:32,567] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:36:32,567] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:36:32,567] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.424s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_4'}
[2022-09-10 17:36:32,666] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:32,666] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:33,041] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:33,071] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:36:33,149] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:36:33,149] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-10 17:36:33,150] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:36:33,699] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:36:33,727] - [task:165] INFO -  {'uptime': '325', 'memoryTotal': 15466930176, 'memoryFree': 13147492352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:36:33,752] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:36:33,752] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:36:33,753] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:36:33,787] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:36:33,820] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:36:33,820] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:36:33,848] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:36:33,850] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:36:33,850] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:36:33,850] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:36:33,898] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:36:33,901] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:33,901] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:34,260] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:34,261] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:36:34,341] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:36:34,342] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:36:34,376] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:34,403] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:34,431] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:36:34,552] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:36:34,580] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13069123584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:36:34,607] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:36:34,637] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:36:34,637] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:36:34,688] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:36:34,691] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:34,691] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:35,074] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:35,075] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:36:35,157] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:36:35,158] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:36:35,189] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:35,215] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:35,244] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:36:35,369] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:36:35,397] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13152260096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:36:35,427] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:36:35,458] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:36:35,458] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:36:35,510] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:36:35,513] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:35,514] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:35,947] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:35,949] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:36:36,032] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:36:36,033] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:36:36,063] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:36,089] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:36,117] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:36:36,233] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:36:36,259] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13150814208, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:36:36,286] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:36:36,315] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:36:36,315] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:36:36,364] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:36:36,368] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:36,368] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:36,721] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:36,722] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:36:36,806] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:36:36,807] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:36:36,836] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:36,862] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:36:36,890] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:36:36,986] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:36:37,496] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:36:42,501] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:36:42,587] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:36:42,591] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:42,591] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:43,007] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:43,009] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:36:43,100] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:36:43,101] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:36:43,101] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:36:44,100] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:36:44,152] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 17:36:44,153] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:36:45,031] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:36:45,343] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:36:45,624] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:36:45,626] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:36:45,684] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:45,684] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:46,061] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:46,065] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:46,065] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:46,409] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:46,413] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:46,413] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:46,875] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:46,889] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:46,889] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:47,549] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:52,271] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:36:52,272] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.2315344467808, 'mem_free': 13148569600, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:36:52,272] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:36:52,272] - [basetestcase:467] INFO - Time to execute basesetup : 19.60883116722107
[2022-09-10 17:36:52,326] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:36:52,326] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:36:52,383] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:36:52,384] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:36:52,442] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:36:52,443] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:36:52,500] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:36:52,500] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:36:52,555] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:36:52,621] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:36:52,622] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:36:52,622] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:36:57,632] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:36:57,636] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:36:57,636] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:36:58,029] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:36:59,004] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:36:59,256] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:37:01,765] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:37:01,781] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 17:37:02,468] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:37:02,469] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:37:02,469] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:37:32,497] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:37:32,527] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:37:32,554] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:37:32,624] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.936127ms
[2022-09-10 17:37:32,624] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '690cb289-0bb0-49d4-a24a-fe90bc9b559b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.936127ms', 'executionTime': '67.821992ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:37:32,624] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:37:32,651] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:37:32,677] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:37:33,364] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 685.032696ms
[2022-09-10 17:37:33,364] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:37:33,417] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:37:33,462] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:37:33,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.844177ms
[2022-09-10 17:37:33,684] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:37:33,722] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:37:33,745] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:37:33,745] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:37:33,760] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:37:33,838] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:37:34,637] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec1d6289d1df84ec5b0b92355b6ee9275job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:37:34,663] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec1d6289d1df84ec5b0b92355b6ee9275job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:37:34,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.950375ms
[2022-09-10 17:37:34,732] - [base_gsi:282] INFO - BUILD INDEX on default(employeec1d6289d1df84ec5b0b92355b6ee9275job_title) USING GSI
[2022-09-10 17:37:35,763] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec1d6289d1df84ec5b0b92355b6ee9275job_title) USING GSI
[2022-09-10 17:37:35,788] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec1d6289d1df84ec5b0b92355b6ee9275job_title%29+USING+GSI
[2022-09-10 17:37:35,816] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.182585ms
[2022-09-10 17:37:36,847] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1d6289d1df84ec5b0b92355b6ee9275job_title'
[2022-09-10 17:37:36,872] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1d6289d1df84ec5b0b92355b6ee9275job_title%27
[2022-09-10 17:37:36,882] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.31443ms
[2022-09-10 17:37:37,914] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1d6289d1df84ec5b0b92355b6ee9275job_title'
[2022-09-10 17:37:37,941] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1d6289d1df84ec5b0b92355b6ee9275job_title%27
[2022-09-10 17:37:37,952] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.790287ms
[2022-09-10 17:37:38,426] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 17:37:38,590] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:37:39,462] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:37:39,654] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-10 17:37:39,654] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:37:39,837] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-10 17:37:39,837] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-10 17:37:39,838] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2690085181, 0, b'')
[2022-09-10 17:37:39,953] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:37:39,983] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:37:40,014] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 17:37:40,018] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.687694ms
[2022-09-10 17:37:40,019] - [task:3246] INFO - {'requestID': '5c49e65a-e8ff-4312-9026-c50dac132afd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec1d6289d1df84ec5b0b92355b6ee9275job_title', 'index_id': '123a30a61e06afc6', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.687694ms', 'executionTime': '2.62905ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 17:37:40,019] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:37:40,019] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:37:40,020] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:37:40,020] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:37:40,020] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 17:37:40,021] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:37:40,021] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:37:40,021] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:37:40,021] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:37:41,021] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:37:41,052] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:37:41,077] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 17:37:41,265] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 179.330965ms
[2022-09-10 17:37:41,265] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:37:41,266] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:37:42,059] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:37:42,060] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:37:43,092] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1d6289d1df84ec5b0b92355b6ee9275job_title'
[2022-09-10 17:37:43,120] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1d6289d1df84ec5b0b92355b6ee9275job_title%27
[2022-09-10 17:37:43,131] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.033451ms
[2022-09-10 17:37:43,158] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec1d6289d1df84ec5b0b92355b6ee9275job_title ON default USING GSI
[2022-09-10 17:37:43,186] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec1d6289d1df84ec5b0b92355b6ee9275job_title+ON+default+USING+GSI
[2022-09-10 17:37:43,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.466001ms
[2022-09-10 17:37:43,268] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1d6289d1df84ec5b0b92355b6ee9275job_title'
[2022-09-10 17:37:43,301] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1d6289d1df84ec5b0b92355b6ee9275job_title%27
[2022-09-10 17:37:43,308] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.861966ms
[2022-09-10 17:37:43,309] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0041d132-5e29-4b26-8e51-cf8fa0b59bc7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.861966ms', 'executionTime': '5.803843ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:37:43,421] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:37:43,424] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:43,424] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:43,863] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:43,918] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:37:43,919] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:37:43,998] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:37:44,054] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:37:44,057] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:44,057] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:44,493] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:44,551] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:37:44,551] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:37:44,630] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:37:44,691] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:37:44,691] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:37:44,691] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:37:44,719] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:37:44,745] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:37:44,755] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.149175ms
[2022-09-10 17:37:44,782] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:37:44,809] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:37:44,865] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.142401ms
[2022-09-10 17:37:44,938] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:37:44,938] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.04864680442557, 'mem_free': 12998533120, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:37:44,939] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:37:44,943] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:44,943] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:45,424] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:45,429] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:45,429] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:46,182] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:46,193] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:46,194] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:46,943] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:46,950] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:46,951] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:47,698] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:52,127] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:37:52,273] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:37:52,642] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:37:52,670] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:37:52,670] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:37:52,727] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:37:52,788] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:37:52,843] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:37:52,844] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:37:52,928] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:37:52,929] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:37:52,956] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:37:53,097] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:37:53,097] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:37:53,126] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:37:53,152] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:37:53,153] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:37:53,182] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:37:53,208] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:37:53,209] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:37:53,237] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:37:53,264] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:37:53,265] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:37:53,293] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:37:53,293] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:37:53,293] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:37:53,294] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.690s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_5'}
[2022-09-10 17:37:53,417] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:53,417] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:53,885] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:53,919] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:37:54,008] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:37:54,008] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-10 17:37:54,009] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:37:54,442] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:37:54,468] - [task:165] INFO -  {'uptime': '406', 'memoryTotal': 15466930176, 'memoryFree': 13108531200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:37:54,495] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:37:54,495] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:37:54,496] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:37:54,524] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:37:54,560] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:37:54,561] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:37:54,590] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:37:54,591] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:37:54,591] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:37:54,591] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:37:54,639] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:37:54,642] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:54,642] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:55,071] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:55,072] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:37:55,159] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:37:55,160] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:37:55,190] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:55,215] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:55,243] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:37:55,364] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:37:55,391] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13000822784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:37:55,417] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:37:55,444] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:37:55,444] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:37:55,493] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:37:55,497] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:55,497] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:55,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:55,910] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:37:56,000] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:37:56,001] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:37:56,031] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:56,056] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:56,083] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:37:56,200] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:37:56,227] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13108195328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:37:56,253] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:37:56,280] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:37:56,281] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:37:56,331] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:37:56,336] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:56,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:56,736] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:56,737] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:37:56,823] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:37:56,824] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:37:56,855] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:56,880] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:56,909] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:37:57,033] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:37:57,060] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13105356800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:37:57,086] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:37:57,114] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:37:57,114] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:37:57,169] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:37:57,173] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:37:57,173] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:37:57,617] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:37:57,618] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:37:57,716] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:37:57,718] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:37:57,748] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:57,774] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:37:57,802] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:37:57,894] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:37:58,556] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:38:03,558] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:38:03,642] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:38:03,645] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:38:03,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:38:04,098] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:38:04,099] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:38:04,187] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:38:04,188] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:38:04,188] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:38:05,016] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:38:05,073] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 17:38:05,075] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:38:05,971] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:38:06,270] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:38:06,522] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:38:06,525] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:38:06,579] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:38:06,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:38:06,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:38:06,990] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:38:06,990] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:38:07,389] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:38:07,394] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:38:07,394] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:38:08,073] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:38:08,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:38:08,081] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:38:08,881] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:38:14,244] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:38:14,244] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.381200941248, 'mem_free': 13108076544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:38:14,244] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:38:14,245] - [basetestcase:467] INFO - Time to execute basesetup : 20.83109426498413
[2022-09-10 17:38:14,303] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:38:14,303] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:38:14,360] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:38:14,360] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:38:14,415] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:38:14,416] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:38:14,469] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:38:14,470] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:38:14,522] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:38:14,589] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:38:14,589] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:38:14,589] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:38:19,600] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:38:19,604] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:38:19,604] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:38:20,032] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:38:21,027] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:38:21,275] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:38:23,717] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:38:23,735] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 17:38:24,821] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:38:24,821] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:38:24,821] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:38:54,845] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:38:54,876] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:38:54,904] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:38:54,982] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 76.012081ms
[2022-09-10 17:38:54,983] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6374a72b-3996-4e87-9fbf-4959085bd6a8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '76.012081ms', 'executionTime': '75.931649ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:38:54,983] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:38:55,012] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:38:55,038] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:38:55,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 690.967469ms
[2022-09-10 17:38:55,732] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:38:55,793] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:38:55,846] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:38:55,857] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.864892ms
[2022-09-10 17:38:56,056] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:38:56,090] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:38:56,107] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:38:56,107] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:38:56,121] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:38:56,207] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:38:57,011] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee06979972e4b54af98535cceb59931c4cjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:38:57,038] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee06979972e4b54af98535cceb59931c4cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:38:57,101] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 59.428819ms
[2022-09-10 17:38:57,101] - [base_gsi:282] INFO - BUILD INDEX on default(employee06979972e4b54af98535cceb59931c4cjob_title) USING GSI
[2022-09-10 17:38:58,133] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee06979972e4b54af98535cceb59931c4cjob_title) USING GSI
[2022-09-10 17:38:58,160] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee06979972e4b54af98535cceb59931c4cjob_title%29+USING+GSI
[2022-09-10 17:38:58,182] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.277201ms
[2022-09-10 17:38:59,212] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06979972e4b54af98535cceb59931c4cjob_title'
[2022-09-10 17:38:59,238] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06979972e4b54af98535cceb59931c4cjob_title%27
[2022-09-10 17:38:59,247] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.072524ms
[2022-09-10 17:39:00,278] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06979972e4b54af98535cceb59931c4cjob_title'
[2022-09-10 17:39:00,304] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06979972e4b54af98535cceb59931c4cjob_title%27
[2022-09-10 17:39:00,312] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.266637ms
[2022-09-10 17:39:00,798] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:39:00,969] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:39:04,595] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:39:05,316] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:39:05,349] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:39:05,377] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 17:39:05,380] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.059267ms
[2022-09-10 17:39:05,381] - [task:3246] INFO - {'requestID': '97d91bda-8cb4-4b7f-9e5c-0152716d4f2f', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee06979972e4b54af98535cceb59931c4cjob_title', 'index_id': '9174988ac4b40a1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.059267ms', 'executionTime': '1.999761ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-09-10 17:39:05,381] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:39:05,381] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:39:05,382] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:39:05,382] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:39:05,383] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 17:39:05,383] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:39:05,383] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:39:05,383] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:39:05,383] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:39:06,383] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:39:06,414] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:39:06,439] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 17:39:06,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 149.241515ms
[2022-09-10 17:39:06,605] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:39:06,605] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:39:08,150] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:39:08,150] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:39:09,182] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06979972e4b54af98535cceb59931c4cjob_title'
[2022-09-10 17:39:09,209] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06979972e4b54af98535cceb59931c4cjob_title%27
[2022-09-10 17:39:09,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.985827ms
[2022-09-10 17:39:09,247] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee06979972e4b54af98535cceb59931c4cjob_title ON default USING GSI
[2022-09-10 17:39:09,274] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee06979972e4b54af98535cceb59931c4cjob_title+ON+default+USING+GSI
[2022-09-10 17:39:09,333] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.553178ms
[2022-09-10 17:39:09,369] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06979972e4b54af98535cceb59931c4cjob_title'
[2022-09-10 17:39:09,400] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06979972e4b54af98535cceb59931c4cjob_title%27
[2022-09-10 17:39:09,408] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.326815ms
[2022-09-10 17:39:09,408] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6ee62421-44e9-4921-bc40-c16823fc3e9b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.326815ms', 'executionTime': '6.267761ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:39:09,519] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:39:09,522] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:09,522] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:10,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:10,143] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:39:10,144] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:39:10,231] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:39:10,287] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:39:10,290] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:10,290] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:10,816] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:10,872] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:39:10,872] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:39:10,959] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:39:11,014] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:39:11,015] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:39:11,015] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:39:11,042] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:39:11,068] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:39:11,078] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.685803ms
[2022-09-10 17:39:11,107] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:39:11,132] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:39:11,190] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.444971ms
[2022-09-10 17:39:11,258] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:39:11,258] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.08643897680546, 'mem_free': 12929150976, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:39:11,258] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:39:11,262] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:11,262] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:11,758] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:11,763] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:11,763] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:12,259] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:12,264] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:12,264] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:13,011] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:13,020] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:13,020] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:13,963] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:20,264] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:39:20,412] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:39:21,597] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:39:21,625] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:39:21,625] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:39:21,685] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:39:21,739] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:39:21,791] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:39:21,792] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:39:21,870] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:39:21,871] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:39:21,898] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:39:22,031] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:39:22,031] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:39:22,058] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:39:22,084] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:39:22,084] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:39:22,111] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:39:22,137] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:39:22,137] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:39:22,164] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:39:22,190] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:39:22,191] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:39:22,218] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:39:22,218] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:39:22,218] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:39:22,219] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 88.859s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_6'}
[2022-09-10 17:39:22,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:22,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:22,859] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:22,892] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:39:22,975] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:39:22,975] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-10 17:39:22,976] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:39:23,351] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:39:23,380] - [task:165] INFO -  {'uptime': '495', 'memoryTotal': 15466930176, 'memoryFree': 13049131008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:39:23,406] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:39:23,406] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:39:23,407] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:39:23,436] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:39:23,475] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:39:23,476] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:39:23,506] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:39:23,507] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:39:23,507] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:39:23,507] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:39:23,556] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:39:23,559] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:23,559] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:24,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:24,107] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:39:24,217] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:39:24,218] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:39:24,248] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:24,275] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:24,307] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:39:24,433] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:39:24,464] - [task:165] INFO -  {'uptime': '492', 'memoryTotal': 15466930176, 'memoryFree': 13049597952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:39:24,493] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:39:24,523] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:39:24,524] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:39:24,576] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:39:24,579] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:24,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:25,139] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:25,140] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:39:25,247] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:39:25,248] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:39:25,279] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:25,305] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:25,333] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:39:25,454] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:39:25,483] - [task:165] INFO -  {'uptime': '493', 'memoryTotal': 15466930176, 'memoryFree': 13053927424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:39:25,509] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:39:25,540] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:39:25,541] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:39:25,592] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:39:25,596] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:25,596] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:26,095] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:26,096] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:39:26,210] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:39:26,211] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:39:26,241] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:26,268] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:26,296] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:39:26,410] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:39:26,436] - [task:165] INFO -  {'uptime': '492', 'memoryTotal': 15466930176, 'memoryFree': 13055741952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:39:26,461] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:39:26,490] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:39:26,490] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:39:26,539] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:39:26,543] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:26,543] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:27,040] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:27,041] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:39:27,142] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:39:27,143] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:39:27,173] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:27,199] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:39:27,227] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:39:27,314] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:39:27,837] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:39:32,842] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:39:32,927] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:39:32,932] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:32,932] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:33,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:33,462] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:39:33,568] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:39:33,569] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:39:33,569] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:39:34,481] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:39:34,535] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 17:39:34,536] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:39:35,436] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:39:35,746] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:39:36,007] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:39:36,010] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:39:36,066] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:36,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:36,598] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:36,602] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:36,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:37,230] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:37,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:37,237] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:38,118] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:38,126] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:38,127] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:38,995] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:44,644] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:39:44,644] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.2538226611636, 'mem_free': 13053243392, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:39:44,644] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:39:44,644] - [basetestcase:467] INFO - Time to execute basesetup : 22.322805166244507
[2022-09-10 17:39:44,699] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:39:44,699] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:39:44,754] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:39:44,754] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:39:44,806] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:39:44,806] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:39:44,858] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:39:44,859] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:39:44,909] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:39:44,966] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:39:44,967] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:39:44,967] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:39:49,980] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:39:49,984] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:39:49,984] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:39:50,531] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:39:51,536] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:39:51,807] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:39:54,040] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:39:54,058] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 17:39:55,466] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:39:55,466] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:39:55,467] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:40:25,494] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:40:25,525] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:40:25,551] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:40:25,619] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.41168ms
[2022-09-10 17:40:25,619] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1c0a4f07-6a4d-4a23-ae9b-95715c058703', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.41168ms', 'executionTime': '65.333113ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:40:25,619] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:40:25,645] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:40:25,670] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:40:26,373] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 701.042514ms
[2022-09-10 17:40:26,374] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:40:26,447] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:40:26,481] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:40:26,489] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.901692ms
[2022-09-10 17:40:26,704] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:40:26,746] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:40:26,761] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:40:26,762] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:40:26,782] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:40:26,863] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:40:27,659] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee3cb715af6da4e6fa292558097e5a1a0job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:40:27,685] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee3cb715af6da4e6fa292558097e5a1a0job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:40:27,737] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.946552ms
[2022-09-10 17:40:27,738] - [base_gsi:282] INFO - BUILD INDEX on default(employeee3cb715af6da4e6fa292558097e5a1a0job_title) USING GSI
[2022-09-10 17:40:28,768] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee3cb715af6da4e6fa292558097e5a1a0job_title) USING GSI
[2022-09-10 17:40:28,794] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee3cb715af6da4e6fa292558097e5a1a0job_title%29+USING+GSI
[2022-09-10 17:40:28,826] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.685643ms
[2022-09-10 17:40:29,856] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee3cb715af6da4e6fa292558097e5a1a0job_title'
[2022-09-10 17:40:29,882] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee3cb715af6da4e6fa292558097e5a1a0job_title%27
[2022-09-10 17:40:29,892] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.840812ms
[2022-09-10 17:40:30,922] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee3cb715af6da4e6fa292558097e5a1a0job_title'
[2022-09-10 17:40:30,949] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee3cb715af6da4e6fa292558097e5a1a0job_title%27
[2022-09-10 17:40:30,956] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.461517ms
[2022-09-10 17:40:31,459] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 17:40:31,624] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:40:32,706] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:40:33,333] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 17:40:33,509] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:40:34,823] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:40:35,229] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:40:35,429] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:40:39,076] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:40:39,977] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:40:40,008] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:40:40,034] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 17:40:40,039] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.792235ms
[2022-09-10 17:40:40,039] - [task:3246] INFO - {'requestID': '3d3aa767-2937-4042-99dc-1cb3c2422436', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee3cb715af6da4e6fa292558097e5a1a0job_title', 'index_id': 'bb9e12ea5f34d9aa', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.792235ms', 'executionTime': '2.71827ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 17:40:40,039] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:40:40,040] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:40:40,040] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:40:40,041] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:40:40,041] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 17:40:40,041] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:40:40,041] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:40:40,042] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:40:40,042] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 17:40:41,041] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:40:41,071] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 17:40:41,098] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 17:40:41,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 151.8126ms
[2022-09-10 17:40:41,472] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:40:41,473] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:40:43,049] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:40:43,050] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 17:40:44,080] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee3cb715af6da4e6fa292558097e5a1a0job_title'
[2022-09-10 17:40:44,107] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee3cb715af6da4e6fa292558097e5a1a0job_title%27
[2022-09-10 17:40:44,116] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.559551ms
[2022-09-10 17:40:44,142] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee3cb715af6da4e6fa292558097e5a1a0job_title ON default USING GSI
[2022-09-10 17:40:44,167] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee3cb715af6da4e6fa292558097e5a1a0job_title+ON+default+USING+GSI
[2022-09-10 17:40:44,206] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.977284ms
[2022-09-10 17:40:44,242] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee3cb715af6da4e6fa292558097e5a1a0job_title'
[2022-09-10 17:40:44,268] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee3cb715af6da4e6fa292558097e5a1a0job_title%27
[2022-09-10 17:40:44,275] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.371424ms
[2022-09-10 17:40:44,275] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7d323c5d-d83f-4cc0-b9e5-7e5803f92d56', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.371424ms', 'executionTime': '5.305625ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:40:44,381] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:40:44,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:40:44,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:40:45,066] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:40:45,129] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:40:45,129] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:40:45,235] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:40:45,296] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:40:45,299] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:40:45,299] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:40:45,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:40:46,045] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:40:46,046] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:40:46,155] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:40:46,212] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:40:46,212] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:40:46,212] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:40:46,239] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:40:46,265] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:40:46,275] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.827568ms
[2022-09-10 17:40:46,304] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:40:46,329] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:40:46,387] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.250374ms
[2022-09-10 17:40:46,458] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:40:46,458] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.73262314549019, 'mem_free': 12891512832, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:40:46,458] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:40:46,462] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:40:46,462] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:40:47,142] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:40:47,148] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:40:47,148] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:40:47,784] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:40:47,789] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:40:47,789] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:40:48,709] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:40:48,720] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:40:48,720] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:40:49,874] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:40:57,627] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:40:57,774] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:40:59,058] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:40:59,089] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:40:59,090] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:40:59,157] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:40:59,212] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:40:59,265] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:40:59,266] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:40:59,347] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:40:59,348] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:40:59,374] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:40:59,509] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:40:59,509] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:40:59,538] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:40:59,565] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:40:59,565] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:40:59,593] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:40:59,619] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:40:59,619] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:40:59,646] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:40:59,672] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:40:59,673] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:40:59,701] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:40:59,701] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 17:40:59,701] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:40:59,702] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 97.437s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_7'}
[2022-09-10 17:40:59,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:40:59,840] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:00,481] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:00,516] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:41:00,600] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:41:00,600] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-10 17:41:00,601] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:41:00,867] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:41:00,896] - [task:165] INFO -  {'uptime': '592', 'memoryTotal': 15466930176, 'memoryFree': 12980912128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:41:00,923] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:41:00,923] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:41:00,924] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:41:00,958] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:41:00,999] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:41:01,000] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:41:01,029] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:41:01,029] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:41:01,030] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:41:01,030] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:41:01,078] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:41:01,082] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:01,082] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:01,685] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:01,686] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:41:01,806] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:41:01,807] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:41:01,839] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:01,865] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:01,893] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:41:02,018] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:41:02,045] - [task:165] INFO -  {'uptime': '592', 'memoryTotal': 15466930176, 'memoryFree': 12983341056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:41:02,070] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:41:02,098] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:41:02,099] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:41:02,150] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:41:02,153] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:02,153] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:02,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:02,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:41:02,885] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:41:02,886] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:41:02,917] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:02,942] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:02,971] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:41:03,092] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:41:03,119] - [task:165] INFO -  {'uptime': '588', 'memoryTotal': 15466930176, 'memoryFree': 12983316480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:41:03,144] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:41:03,172] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:41:03,172] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:41:03,224] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:41:03,228] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:03,229] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:03,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:03,810] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:41:03,923] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:41:03,924] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:41:03,956] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:03,981] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:04,010] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:41:04,134] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:41:04,161] - [task:165] INFO -  {'uptime': '588', 'memoryTotal': 15466930176, 'memoryFree': 12982874112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:41:04,186] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:41:04,213] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:41:04,214] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:41:04,263] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:41:04,266] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:04,266] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:04,884] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:04,885] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:41:05,008] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:41:05,009] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:41:05,042] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:05,071] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:41:05,101] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:41:05,191] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:41:05,808] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:41:10,814] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:41:10,898] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:41:10,903] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:10,903] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:11,506] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:11,507] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:41:11,630] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:41:11,631] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:41:11,632] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:41:12,342] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:41:12,395] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 17:41:12,395] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:41:13,340] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:41:13,638] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:41:13,900] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:41:13,903] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 17:41:13,960] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:13,960] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:14,582] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:14,587] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:14,587] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:15,496] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:15,504] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:15,504] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:16,602] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:16,613] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:16,613] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:17,712] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:24,125] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:41:24,126] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.03339987910768, 'mem_free': 12995727360, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:41:24,126] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:41:24,126] - [basetestcase:467] INFO - Time to execute basesetup : 24.288872957229614
[2022-09-10 17:41:24,177] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:41:24,177] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:41:24,237] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:41:24,237] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:41:24,290] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:41:24,290] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:41:24,342] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:41:24,342] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:41:24,394] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:41:24,449] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:41:24,450] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:41:24,450] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:41:29,462] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:41:29,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:41:29,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:41:30,097] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:41:31,147] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:41:31,323] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:41:33,495] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:41:33,580] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:41:33,580] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:41:33,580] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:42:03,609] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:42:03,639] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:42:03,665] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:42:03,739] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.72851ms
[2022-09-10 17:42:03,739] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3260ccec-81b4-4dbe-8844-e17b82aca4e9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.72851ms', 'executionTime': '70.661787ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:42:03,739] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:42:03,768] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:42:03,794] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:42:04,472] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 676.380382ms
[2022-09-10 17:42:04,473] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:42:04,533] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:42:04,571] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:42:04,580] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.436376ms
[2022-09-10 17:42:04,784] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:42:04,843] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:42:04,864] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:42:04,864] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:42:04,885] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:42:04,886] - [base_gsi:326] INFO - []
[2022-09-10 17:42:05,753] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee10ebaef90f1c4eeb9e28decfb1b60c19job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:42:05,778] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee10ebaef90f1c4eeb9e28decfb1b60c19job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:42:05,854] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 73.893432ms
[2022-09-10 17:42:05,890] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:42:05,917] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:42:05,977] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.484393ms
[2022-09-10 17:42:05,978] - [base_gsi:282] INFO - BUILD INDEX on default(employee10ebaef90f1c4eeb9e28decfb1b60c19job_title,employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr) USING GSI
[2022-09-10 17:42:07,009] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee10ebaef90f1c4eeb9e28decfb1b60c19job_title,employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr) USING GSI
[2022-09-10 17:42:07,035] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee10ebaef90f1c4eeb9e28decfb1b60c19job_title%2Cemployee10ebaef90f1c4eeb9e28decfb1b60c19join_yr%29+USING+GSI
[2022-09-10 17:42:07,077] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.607558ms
[2022-09-10 17:42:08,107] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee10ebaef90f1c4eeb9e28decfb1b60c19job_title'
[2022-09-10 17:42:08,132] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee10ebaef90f1c4eeb9e28decfb1b60c19job_title%27
[2022-09-10 17:42:08,141] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.460187ms
[2022-09-10 17:42:09,173] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee10ebaef90f1c4eeb9e28decfb1b60c19job_title'
[2022-09-10 17:42:09,199] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee10ebaef90f1c4eeb9e28decfb1b60c19job_title%27
[2022-09-10 17:42:09,208] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.72789ms
[2022-09-10 17:42:09,234] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr'
[2022-09-10 17:42:09,261] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr%27
[2022-09-10 17:42:09,264] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.488303ms
[2022-09-10 17:42:10,295] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee10ebaef90f1c4eeb9e28decfb1b60c19job_title'
[2022-09-10 17:42:10,323] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee10ebaef90f1c4eeb9e28decfb1b60c19job_title%27
[2022-09-10 17:42:10,331] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.680786ms
[2022-09-10 17:42:10,359] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee10ebaef90f1c4eeb9e28decfb1b60c19job_title ON default USING GSI
[2022-09-10 17:42:10,386] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee10ebaef90f1c4eeb9e28decfb1b60c19job_title+ON+default+USING+GSI
[2022-09-10 17:42:10,431] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.38367ms
[2022-09-10 17:42:10,474] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr'
[2022-09-10 17:42:10,500] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr%27
[2022-09-10 17:42:10,508] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.744785ms
[2022-09-10 17:42:10,536] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr ON default USING GSI
[2022-09-10 17:42:10,561] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr+ON+default+USING+GSI
[2022-09-10 17:42:10,614] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.136403ms
[2022-09-10 17:42:10,647] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee10ebaef90f1c4eeb9e28decfb1b60c19job_title'
[2022-09-10 17:42:10,675] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee10ebaef90f1c4eeb9e28decfb1b60c19job_title%27
[2022-09-10 17:42:10,683] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.077618ms
[2022-09-10 17:42:10,683] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bb8b68f4-53cc-4d08-8f81-626a087edeb1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.077618ms', 'executionTime': '7.006166ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:42:10,711] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr'
[2022-09-10 17:42:10,737] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee10ebaef90f1c4eeb9e28decfb1b60c19join_yr%27
[2022-09-10 17:42:10,741] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.725623ms
[2022-09-10 17:42:10,741] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4a62a69f-3fb7-40a6-812c-95a7e59d77b5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.725623ms', 'executionTime': '1.656287ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:42:10,851] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:42:10,854] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:10,854] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:11,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:11,519] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:42:11,520] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:42:11,623] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:42:11,681] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:42:11,685] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:11,685] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:12,322] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:12,378] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:42:12,378] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:42:12,484] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:42:12,544] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:42:12,545] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:42:12,545] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:42:12,571] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:42:12,598] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:42:12,607] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.243775ms
[2022-09-10 17:42:12,633] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:42:12,660] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:42:12,721] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.810848ms
[2022-09-10 17:42:12,793] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:42:12,794] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.0178486305388, 'mem_free': 12865757184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:42:12,794] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:42:12,799] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:12,799] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:13,403] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:13,408] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:13,408] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:14,017] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:14,023] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:14,023] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:14,879] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:14,890] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:14,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:15,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:22,959] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-10 17:42:23,106] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:42:23,887] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:42:23,915] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:42:23,915] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:42:23,977] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:42:24,034] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:42:24,086] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:42:24,088] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:42:24,169] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:42:24,171] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:42:24,197] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:42:24,345] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:42:24,346] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:42:24,374] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:42:24,399] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:42:24,399] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:42:24,425] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:42:24,452] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:42:24,453] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:42:24,485] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:42:24,512] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:42:24,512] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:42:24,539] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:42:24,539] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 17:42:24,539] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:42:24,540] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.757s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_8'}
[2022-09-10 17:42:24,622] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:24,623] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:25,255] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:25,287] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:42:25,368] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:42:25,368] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-10 17:42:25,369] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:42:25,652] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:42:25,679] - [task:165] INFO -  {'uptime': '677', 'memoryTotal': 15466930176, 'memoryFree': 13001814016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:42:25,707] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:42:25,707] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:42:25,707] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:42:25,744] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:42:25,783] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:42:25,783] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:42:25,812] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:42:25,813] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:42:25,814] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:42:25,814] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:42:25,867] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:42:25,870] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:25,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:26,505] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:26,506] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:42:26,626] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:42:26,627] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:42:26,658] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:26,685] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:26,715] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:42:26,838] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:42:26,865] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 12886843392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:42:26,892] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:42:26,920] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:42:26,921] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:42:26,977] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:42:26,980] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:26,981] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:27,626] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:27,627] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:42:27,753] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:42:27,755] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:42:27,784] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:27,810] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:27,838] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:42:27,958] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:42:27,985] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 13001728000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:42:28,012] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:42:28,040] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:42:28,041] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:42:28,093] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:42:28,098] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:28,098] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:28,732] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:28,734] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:42:28,857] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:42:28,859] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:42:28,890] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:28,919] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:28,948] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:42:29,071] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:42:29,098] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 13001220096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:42:29,125] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:42:29,154] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:42:29,155] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:42:29,205] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:42:29,208] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:29,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:29,843] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:29,844] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:42:29,968] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:42:29,969] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:42:30,000] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:30,026] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:42:30,055] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:42:30,147] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:42:30,672] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:42:35,674] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:42:35,760] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:42:35,764] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:35,764] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:36,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:36,379] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:42:36,493] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:42:36,494] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:42:36,494] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:42:37,305] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:42:37,361] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 17:42:37,363] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:42:38,416] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:42:38,728] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:42:39,025] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:42:39,029] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 17:42:39,082] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:39,083] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:39,668] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:39,673] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:39,673] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:40,466] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:40,479] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:40,479] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:41,564] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:41,572] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:41,573] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:42,689] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:49,204] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:42:49,205] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.23560029056221, 'mem_free': 12999401472, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:42:49,205] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:42:49,205] - [basetestcase:467] INFO - Time to execute basesetup : 24.58523726463318
[2022-09-10 17:42:49,259] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:42:49,259] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:42:49,317] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:42:49,317] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:42:49,373] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:42:49,373] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:42:49,428] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:42:49,428] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:42:49,480] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:42:49,544] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:42:49,544] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:42:49,544] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:42:54,556] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:42:54,559] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:42:54,559] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:42:55,200] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:42:56,239] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:42:56,522] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:42:58,604] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:42:58,684] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:42:58,684] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:42:58,685] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:43:28,704] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:43:28,733] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:43:28,758] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:43:28,829] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.971678ms
[2022-09-10 17:43:28,830] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '07433c97-fcb1-4096-be97-8aab717e5ba4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.971678ms', 'executionTime': '68.902086ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:43:28,830] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:43:28,855] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:43:28,880] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:43:29,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 703.796937ms
[2022-09-10 17:43:29,586] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:43:29,639] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:43:29,684] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:43:29,692] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.800607ms
[2022-09-10 17:43:29,906] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:43:29,940] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:43:29,961] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:43:29,962] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:43:29,980] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:43:29,981] - [base_gsi:326] INFO - []
[2022-09-10 17:43:30,873] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:43:30,900] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:43:30,969] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.77881ms
[2022-09-10 17:43:30,969] - [base_gsi:282] INFO - BUILD INDEX on default(employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr) USING GSI
[2022-09-10 17:43:32,001] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr) USING GSI
[2022-09-10 17:43:32,028] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr%29+USING+GSI
[2022-09-10 17:43:32,050] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.807645ms
[2022-09-10 17:43:33,081] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr'
[2022-09-10 17:43:33,108] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr%27
[2022-09-10 17:43:33,117] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.838224ms
[2022-09-10 17:43:34,148] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr'
[2022-09-10 17:43:34,174] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr%27
[2022-09-10 17:43:34,183] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.383125ms
[2022-09-10 17:43:35,218] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr'
[2022-09-10 17:43:35,243] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr%27
[2022-09-10 17:43:35,251] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.420995ms
[2022-09-10 17:43:35,276] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr ON default USING GSI
[2022-09-10 17:43:35,301] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr+ON+default+USING+GSI
[2022-09-10 17:43:35,339] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.917804ms
[2022-09-10 17:43:35,373] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr'
[2022-09-10 17:43:35,402] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecfb15dacb714e5395c8ed6c7e360cacjob_title_join_yr%27
[2022-09-10 17:43:35,409] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.93103ms
[2022-09-10 17:43:35,409] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7356e9b6-b8ba-49ae-b229-f68540a9aeae', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.93103ms', 'executionTime': '5.876689ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:43:35,512] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:43:35,515] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:35,515] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:36,154] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:36,211] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:43:36,211] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:43:36,319] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:43:36,378] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:43:36,381] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:36,381] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:37,036] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:37,090] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:43:37,091] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:43:37,203] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:43:37,261] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:43:37,262] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:43:37,262] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:43:37,292] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:43:37,317] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:43:37,328] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.915454ms
[2022-09-10 17:43:37,357] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:43:37,387] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:43:37,435] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.819848ms
[2022-09-10 17:43:37,512] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:43:37,512] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.970945956418935, 'mem_free': 12878827520, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:43:37,512] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:43:37,517] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:37,517] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:38,179] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:38,184] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:38,184] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:39,224] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:39,232] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:39,232] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:40,392] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:40,404] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:40,405] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:41,478] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:48,114] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-10 17:43:48,258] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:43:48,859] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:43:48,887] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:43:48,888] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:43:48,943] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:43:48,995] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:43:49,048] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:43:49,049] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:43:49,127] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:43:49,128] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:43:49,156] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:43:49,291] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:43:49,292] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:43:49,320] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:43:49,347] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:43:49,348] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:43:49,380] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:43:49,419] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:43:49,419] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:43:49,447] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:43:49,473] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:43:49,473] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:43:49,501] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:43:49,501] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 17:43:49,501] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:43:49,502] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.937s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_9'}
[2022-09-10 17:43:49,589] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:49,590] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:50,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:50,286] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:43:50,370] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:43:50,370] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-10 17:43:50,371] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:43:50,616] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:43:50,643] - [task:165] INFO -  {'uptime': '762', 'memoryTotal': 15466930176, 'memoryFree': 12986834944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:43:50,671] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:43:50,671] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:43:50,671] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:43:50,705] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:43:50,742] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:43:50,742] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:43:50,773] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:43:50,773] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:43:50,774] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:43:50,774] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:43:50,825] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:43:50,828] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:50,828] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:51,481] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:51,482] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:43:51,605] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:43:51,606] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:43:51,637] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:51,664] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:51,694] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:43:51,818] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:43:51,845] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 12983898112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:43:51,871] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:43:51,902] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:43:51,902] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:43:51,955] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:43:51,958] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:51,958] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:52,594] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:52,596] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:43:52,716] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:43:52,717] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:43:52,749] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:52,776] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:52,804] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:43:52,921] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:43:52,947] - [task:165] INFO -  {'uptime': '759', 'memoryTotal': 15466930176, 'memoryFree': 12984430592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:43:52,972] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:43:53,002] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:43:53,002] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:43:53,058] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:43:53,063] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:53,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:53,663] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:53,665] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:43:53,782] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:43:53,783] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:43:53,812] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:53,838] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:53,867] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:43:53,990] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:43:54,017] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 12985851904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:43:54,044] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:43:54,073] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:43:54,074] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:43:54,124] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:43:54,127] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:43:54,127] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:43:54,770] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:43:54,771] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:43:54,905] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:43:54,906] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:43:54,938] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:54,963] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:43:54,992] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:43:55,078] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:43:55,612] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:44:00,618] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:44:00,703] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:44:00,707] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:44:00,707] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:44:01,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:44:01,377] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:44:01,507] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:44:01,509] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:44:01,509] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:44:02,194] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:44:02,249] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 17:44:02,249] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:44:03,194] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:44:03,524] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:44:03,844] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:44:03,848] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 17:44:03,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:44:03,915] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:44:04,541] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:44:04,546] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:44:04,546] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:44:05,477] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:44:05,485] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:44:05,486] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:44:06,620] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:44:06,630] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:44:06,631] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:44:07,745] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:44:14,384] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:44:14,384] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.90872219349819, 'mem_free': 12982161408, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:44:14,385] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:44:14,385] - [basetestcase:467] INFO - Time to execute basesetup : 24.797922611236572
[2022-09-10 17:44:14,437] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:44:14,437] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:44:14,492] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:44:14,492] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:44:14,546] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:44:14,546] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:44:14,600] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:44:14,601] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:44:14,654] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:44:14,715] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:44:14,715] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:44:14,715] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:44:19,724] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:44:19,727] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:44:19,727] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:44:20,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:44:21,512] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:44:21,686] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:44:23,807] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:44:23,884] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:44:23,884] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:44:23,885] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:44:53,900] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:44:53,930] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:44:53,962] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:44:54,031] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.197359ms
[2022-09-10 17:44:54,032] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '77f01abe-1d2d-4f9a-a627-766c1267ea2b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.197359ms', 'executionTime': '67.126414ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:44:54,032] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:44:54,059] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:44:54,090] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:44:54,860] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 768.320581ms
[2022-09-10 17:44:54,861] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:44:54,918] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:44:54,966] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:44:54,977] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.707769ms
[2022-09-10 17:44:55,170] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:44:55,214] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:44:55,234] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:44:55,235] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:44:55,268] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:44:55,334] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:44:56,129] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee267fc3adb1174918b04720082f3fce71join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:44:56,155] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee267fc3adb1174918b04720082f3fce71join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:44:56,211] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.868232ms
[2022-09-10 17:44:56,212] - [base_gsi:282] INFO - BUILD INDEX on default(`employee267fc3adb1174918b04720082f3fce71join_yr`) USING GSI
[2022-09-10 17:44:57,242] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee267fc3adb1174918b04720082f3fce71join_yr`) USING GSI
[2022-09-10 17:44:57,270] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee267fc3adb1174918b04720082f3fce71join_yr%60%29+USING+GSI
[2022-09-10 17:44:57,304] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.699521ms
[2022-09-10 17:44:57,349] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee267fc3adb1174918b04720082f3fce71join_yr'
[2022-09-10 17:44:57,379] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee267fc3adb1174918b04720082f3fce71join_yr%27
[2022-09-10 17:44:57,388] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.130156ms
[2022-09-10 17:44:58,418] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee267fc3adb1174918b04720082f3fce71join_yr'
[2022-09-10 17:44:58,444] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee267fc3adb1174918b04720082f3fce71join_yr%27
[2022-09-10 17:44:58,454] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.107488ms
[2022-09-10 17:44:59,485] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee267fc3adb1174918b04720082f3fce71join_yr'
[2022-09-10 17:44:59,511] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee267fc3adb1174918b04720082f3fce71join_yr%27
[2022-09-10 17:44:59,520] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.387374ms
[2022-09-10 17:45:00,552] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:45:00,578] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 17:45:00,583] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.623296ms
[2022-09-10 17:45:00,584] - [base_gsi:504] INFO - {'requestID': '4d739b03-48a5-4c8b-ad72-a48fdbd5cf17', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee267fc3adb1174918b04720082f3fce71join_yr', 'index_id': '4ecac8fcb7b76801', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.623296ms', 'executionTime': '3.555228ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 17:45:00,584] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:45:00,585] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:45:00,585] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:45:00,585] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:45:00,586] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:45:00,587] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:45:00,587] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:45:00,587] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:45:00,647] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:45:00,686] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:45:00,712] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 17:45:00,867] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 141.366282ms
[2022-09-10 17:45:00,867] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:45:00,868] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:45:02,758] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:45:02,786] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee267fc3adb1174918b04720082f3fce71join_yr'
[2022-09-10 17:45:02,812] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee267fc3adb1174918b04720082f3fce71join_yr%27
[2022-09-10 17:45:02,822] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.557172ms
[2022-09-10 17:45:02,822] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b6cc5978-7c16-4c3d-a1b2-0a24455610a5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.557172ms', 'executionTime': '7.498033ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:45:02,931] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:45:02,935] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:02,935] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:03,674] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:03,732] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:45:03,732] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:45:03,848] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:45:03,908] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:45:03,911] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:03,911] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:04,563] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:04,618] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:45:04,619] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:45:04,726] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:45:04,783] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:45:04,783] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:45:04,783] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:45:04,809] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:45:04,834] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:45:04,842] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.519598ms
[2022-09-10 17:45:04,843] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '91cb0ba6-544a-4ccb-9e43-1e8bfef1768b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.519598ms', 'executionTime': '6.375269ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:45:04,898] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:45:04,899] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.68554065169339, 'mem_free': 12834328576, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:45:04,899] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:45:04,903] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:04,903] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:05,545] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:05,550] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:05,550] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:06,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:06,180] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:06,181] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:07,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:07,098] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:07,098] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:08,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:15,860] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-10 17:45:15,999] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:45:16,054] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:45:16,111] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:45:16,167] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:45:16,167] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:45:16,245] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:45:16,246] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:45:16,272] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:45:16,407] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:45:16,407] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:45:16,436] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:45:16,462] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:45:16,463] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:45:16,489] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:45:16,516] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:45:16,516] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:45:16,545] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:45:16,572] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:45:16,572] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:45:16,599] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:45:16,599] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 17:45:16,599] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:45:16,600] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 87.065s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_10'}
[2022-09-10 17:45:16,679] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:16,679] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:17,342] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:17,374] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:45:17,455] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:45:17,455] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-10 17:45:17,456] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:45:17,706] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:45:17,738] - [task:165] INFO -  {'uptime': '846', 'memoryTotal': 15466930176, 'memoryFree': 12986052608, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:45:17,767] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:45:17,767] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:45:17,768] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:45:17,799] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:45:17,835] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:45:17,835] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:45:17,863] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:45:17,864] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:45:17,864] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:45:17,865] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:45:17,914] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:45:17,917] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:17,917] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:18,588] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:18,589] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:45:18,719] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:45:18,720] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:45:18,749] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:18,775] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:18,804] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:45:18,930] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:45:18,960] - [task:165] INFO -  {'uptime': '849', 'memoryTotal': 15466930176, 'memoryFree': 12986175488, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:45:18,987] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:45:19,018] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:45:19,019] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:45:19,070] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:45:19,075] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:19,076] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:19,750] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:19,752] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:45:19,883] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:45:19,884] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:45:19,916] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:19,944] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:19,974] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:45:20,096] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:45:20,123] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 12986044416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:45:20,150] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:45:20,178] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:45:20,179] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:45:20,236] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:45:20,239] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:20,239] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:20,923] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:20,924] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:45:21,052] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:45:21,053] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:45:21,083] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:21,108] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:21,135] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:45:21,250] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:45:21,277] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 12985413632, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:45:21,303] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:45:21,330] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:45:21,330] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:45:21,380] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:45:21,384] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:21,384] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:22,060] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:22,061] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:45:22,191] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:45:22,192] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:45:22,223] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:22,250] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:45:22,280] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:45:22,375] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:45:22,902] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:45:27,905] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:45:27,992] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:45:27,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:27,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:28,659] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:28,660] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:45:28,782] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:45:28,783] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:45:28,783] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:45:29,487] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:45:29,545] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 17:45:29,557] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:45:30,483] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:45:30,802] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:45:31,064] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:45:31,067] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 17:45:31,121] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:31,122] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:31,790] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:31,795] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:31,795] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:32,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:32,828] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:32,828] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:33,977] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:33,989] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:33,989] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:35,132] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:41,956] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:45:41,956] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.87555673819715, 'mem_free': 12983107584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:45:41,957] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:45:41,957] - [basetestcase:467] INFO - Time to execute basesetup : 25.280398845672607
[2022-09-10 17:45:42,012] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:45:42,012] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:45:42,067] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:45:42,068] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:45:42,123] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:45:42,123] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:45:42,178] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:45:42,179] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:45:42,234] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:45:42,308] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:45:42,308] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:45:42,309] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:45:47,320] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:45:47,323] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:45:47,323] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:45:47,986] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:45:49,012] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:45:49,296] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:45:51,413] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:45:51,491] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:45:51,492] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:45:51,492] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:46:21,522] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:46:21,553] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:46:21,579] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:46:21,649] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.898365ms
[2022-09-10 17:46:21,649] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3236ef42-6e2a-49a5-8889-dab7a42166a6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.898365ms', 'executionTime': '67.792345ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:46:21,649] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:46:21,676] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:46:21,702] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:46:22,427] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 723.505117ms
[2022-09-10 17:46:22,428] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:46:22,512] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:46:22,550] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:46:22,560] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.846746ms
[2022-09-10 17:46:22,762] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:46:22,815] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:46:22,838] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:46:22,838] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:46:22,857] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:46:22,925] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:46:23,724] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0c454fc0748e4a1983984f1157f799c2join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:46:23,751] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0c454fc0748e4a1983984f1157f799c2join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:46:23,804] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.239654ms
[2022-09-10 17:46:23,805] - [base_gsi:282] INFO - BUILD INDEX on default(`employee0c454fc0748e4a1983984f1157f799c2join_yr`) USING GSI
[2022-09-10 17:46:24,837] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee0c454fc0748e4a1983984f1157f799c2join_yr`) USING GSI
[2022-09-10 17:46:24,863] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee0c454fc0748e4a1983984f1157f799c2join_yr%60%29+USING+GSI
[2022-09-10 17:46:24,887] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.236135ms
[2022-09-10 17:46:24,936] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0c454fc0748e4a1983984f1157f799c2join_yr'
[2022-09-10 17:46:24,972] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0c454fc0748e4a1983984f1157f799c2join_yr%27
[2022-09-10 17:46:24,980] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.619273ms
[2022-09-10 17:46:26,011] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0c454fc0748e4a1983984f1157f799c2join_yr'
[2022-09-10 17:46:26,041] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0c454fc0748e4a1983984f1157f799c2join_yr%27
[2022-09-10 17:46:26,052] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.334002ms
[2022-09-10 17:46:27,082] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0c454fc0748e4a1983984f1157f799c2join_yr'
[2022-09-10 17:46:27,108] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0c454fc0748e4a1983984f1157f799c2join_yr%27
[2022-09-10 17:46:27,117] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.977442ms
[2022-09-10 17:46:28,146] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:46:28,172] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 17:46:28,176] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.853148ms
[2022-09-10 17:46:28,177] - [base_gsi:504] INFO - {'requestID': '77e9ab47-04d2-4fb9-a4a7-43e0370d70af', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0c454fc0748e4a1983984f1157f799c2join_yr', 'index_id': '71cc6e0743de4f6b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.853148ms', 'executionTime': '2.776879ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 17:46:28,177] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:46:28,177] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:46:28,178] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:46:28,178] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:46:28,178] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:46:28,178] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:46:28,179] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:46:28,179] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:46:28,239] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:46:28,277] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:46:28,302] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 17:46:28,482] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 167.305569ms
[2022-09-10 17:46:28,483] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:46:28,483] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:46:29,666] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-10 17:46:29,717] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-10 17:46:29,748] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:46:29,775] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 17:46:29,779] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.156301ms
[2022-09-10 17:46:29,779] - [base_gsi:504] INFO - {'requestID': '011fde6c-34ec-44f2-a7a3-e4eb1cc1f990', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0c454fc0748e4a1983984f1157f799c2join_yr', 'index_id': '71cc6e0743de4f6b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.156301ms', 'executionTime': '2.065272ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 17:46:29,780] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:46:29,780] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:46:29,780] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:46:29,780] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:46:29,780] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:46:29,781] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:46:29,781] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:46:29,781] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:46:29,835] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:46:29,874] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:46:29,899] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 17:46:30,062] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 154.097169ms
[2022-09-10 17:46:30,062] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:46:30,062] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:46:31,260] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0c454fc0748e4a1983984f1157f799c2join_yr ON default USING GSI
[2022-09-10 17:46:31,286] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0c454fc0748e4a1983984f1157f799c2join_yr+ON+default+USING+GSI
[2022-09-10 17:46:31,329] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.042076ms
[2022-09-10 17:46:31,365] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0c454fc0748e4a1983984f1157f799c2join_yr'
[2022-09-10 17:46:31,391] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0c454fc0748e4a1983984f1157f799c2join_yr%27
[2022-09-10 17:46:31,399] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.030497ms
[2022-09-10 17:46:31,399] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '84ccd4af-f520-44ec-bc29-8bc702775bc9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.030497ms', 'executionTime': '5.930943ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:46:31,503] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:46:31,508] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:31,508] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:32,190] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:32,245] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:46:32,245] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:46:32,355] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:46:32,412] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:46:32,415] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:32,415] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:33,111] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:33,165] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:46:33,165] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:46:33,279] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:46:33,333] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:46:33,334] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:46:33,334] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:46:33,359] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:46:33,384] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:46:33,394] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.170003ms
[2022-09-10 17:46:33,423] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:46:33,447] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:46:33,492] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.079373ms
[2022-09-10 17:46:33,557] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:46:33,558] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.92371423268747, 'mem_free': 12802818048, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:46:33,558] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:46:33,561] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:33,561] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:34,240] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:34,245] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:34,245] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:34,965] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:34,970] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:34,970] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:36,183] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:36,195] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:36,195] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:37,451] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:45,239] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-10 17:46:45,383] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:46:46,042] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:46:46,070] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:46:46,070] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:46:46,127] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:46:46,181] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:46:46,238] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:46:46,239] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:46:46,325] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:46:46,326] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:46:46,354] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:46:46,487] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:46:46,487] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:46:46,526] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:46:46,552] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:46:46,552] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:46:46,580] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:46:46,607] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:46:46,607] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:46:46,635] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:46:46,661] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:46:46,662] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:46:46,688] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:46:46,688] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 17:46:46,688] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:46:46,689] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 90.064s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_11'}
[2022-09-10 17:46:46,768] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:46,768] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:47,476] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:47,509] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:46:47,587] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:46:47,587] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-10 17:46:47,587] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:46:47,796] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:46:47,823] - [task:165] INFO -  {'uptime': '939', 'memoryTotal': 15466930176, 'memoryFree': 12958654464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:46:47,850] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:46:47,851] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:46:47,851] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:46:47,888] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:46:47,921] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:46:47,922] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:46:47,951] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:46:47,952] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:46:47,952] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:46:47,953] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:46:48,002] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:46:48,005] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:48,005] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:48,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:48,694] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:46:48,823] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:46:48,824] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:46:48,855] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:48,880] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:48,909] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:46:49,034] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:46:49,061] - [task:165] INFO -  {'uptime': '939', 'memoryTotal': 15466930176, 'memoryFree': 12961857536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:46:49,086] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:46:49,113] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:46:49,113] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:46:49,164] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:46:49,168] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:49,168] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:49,869] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:49,870] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:46:50,003] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:46:50,004] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:46:50,035] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:50,061] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:50,089] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:46:50,210] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:46:50,239] - [task:165] INFO -  {'uptime': '935', 'memoryTotal': 15466930176, 'memoryFree': 12960911360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:46:50,266] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:46:50,296] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:46:50,297] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:46:50,347] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:46:50,352] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:50,353] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:51,079] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:51,080] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:46:51,212] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:46:51,213] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:46:51,244] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:51,271] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:51,303] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:46:51,424] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:46:51,452] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12963479552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:46:51,480] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:46:51,510] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:46:51,511] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:46:51,564] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:46:51,567] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:51,567] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:52,268] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:52,269] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:46:52,395] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:46:52,396] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:46:52,426] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:52,451] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:46:52,479] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:46:52,571] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:46:53,141] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:46:58,147] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:46:58,239] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:46:58,243] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:46:58,243] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:46:58,970] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:46:58,971] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:46:59,107] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:46:59,108] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:46:59,109] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:46:59,751] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:46:59,807] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 17:46:59,807] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:47:00,725] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:47:01,050] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:47:01,330] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:47:01,333] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 17:47:01,389] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:47:01,389] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:47:02,111] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:47:02,116] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:47:02,116] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:47:03,202] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:47:03,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:47:03,211] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:47:04,441] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:47:04,449] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:47:04,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:47:05,678] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:47:12,790] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:47:12,791] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.42070904915133, 'mem_free': 12958609408, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:47:12,791] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:47:12,791] - [basetestcase:467] INFO - Time to execute basesetup : 26.025761127471924
[2022-09-10 17:47:12,845] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:47:12,846] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:47:12,901] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:47:12,902] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:47:12,953] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:47:12,954] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:47:13,010] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:47:13,010] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:47:13,061] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:47:13,122] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:47:13,122] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:47:13,122] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:47:18,133] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:47:18,136] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:47:18,137] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:47:18,795] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:47:19,954] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:47:20,123] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:47:23,226] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:47:23,310] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:47:23,310] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:47:23,310] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:47:53,320] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:47:53,350] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:47:53,379] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:47:53,452] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 71.541444ms
[2022-09-10 17:47:53,453] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7f996e4f-fb65-4f3f-8a01-fa2c6e8127ff', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '71.541444ms', 'executionTime': '71.459384ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:47:53,453] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:47:53,480] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:47:53,507] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:47:54,179] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 669.419604ms
[2022-09-10 17:47:54,179] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:47:54,250] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:47:54,286] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:47:54,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.200506ms
[2022-09-10 17:47:54,481] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:47:54,514] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:47:54,531] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:47:54,531] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:47:54,552] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:47:54,631] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:47:55,449] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee9f623c04a9b94953b77a8608de842b21join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 17:47:55,477] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee9f623c04a9b94953b77a8608de842b21join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 17:47:55,528] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.347758ms
[2022-09-10 17:47:55,529] - [base_gsi:282] INFO - BUILD INDEX on default(`employee9f623c04a9b94953b77a8608de842b21join_yr`) USING GSI
[2022-09-10 17:47:56,559] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee9f623c04a9b94953b77a8608de842b21join_yr`) USING GSI
[2022-09-10 17:47:56,584] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee9f623c04a9b94953b77a8608de842b21join_yr%60%29+USING+GSI
[2022-09-10 17:47:56,616] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.887845ms
[2022-09-10 17:47:56,656] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9f623c04a9b94953b77a8608de842b21join_yr'
[2022-09-10 17:47:56,690] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9f623c04a9b94953b77a8608de842b21join_yr%27
[2022-09-10 17:47:56,698] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.861979ms
[2022-09-10 17:47:57,730] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9f623c04a9b94953b77a8608de842b21join_yr'
[2022-09-10 17:47:57,757] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9f623c04a9b94953b77a8608de842b21join_yr%27
[2022-09-10 17:47:57,766] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.339622ms
[2022-09-10 17:47:58,798] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9f623c04a9b94953b77a8608de842b21join_yr'
[2022-09-10 17:47:58,826] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9f623c04a9b94953b77a8608de842b21join_yr%27
[2022-09-10 17:47:58,835] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.779217ms
[2022-09-10 17:47:59,867] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:47:59,896] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 17:47:59,900] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.735375ms
[2022-09-10 17:47:59,900] - [base_gsi:504] INFO - {'requestID': '49a406ce-ca5b-47ce-8e3e-e2d8a1f65520', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee9f623c04a9b94953b77a8608de842b21join_yr', 'index_id': '6db7bd56edb2811e', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.735375ms', 'executionTime': '2.659724ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 17:47:59,901] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 17:47:59,901] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:47:59,901] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 17:47:59,901] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 17:47:59,902] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:47:59,902] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:47:59,902] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:47:59,902] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 17:47:59,963] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 17:48:00,005] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 17:48:00,033] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 17:48:00,233] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 186.783675ms
[2022-09-10 17:48:00,234] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 17:48:00,234] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 17:48:02,352] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 17:48:04,988] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:48:05,058] - [on_prem_rest_client:3047] INFO - 0.07 seconds to create bucket default
[2022-09-10 17:48:05,058] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:48:05,958] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:48:06,276] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:48:06,545] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:48:06,548] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 17:48:08,579] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:48:08,579] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:08,607] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:48:08,636] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:48:08,637] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:08,663] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:48:08,691] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:48:08,691] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:08,719] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:48:08,746] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:48:08,746] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:08,773] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:48:08,826] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:48:08,852] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9f623c04a9b94953b77a8608de842b21join_yr'
[2022-09-10 17:48:08,877] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9f623c04a9b94953b77a8608de842b21join_yr%27
[2022-09-10 17:48:08,924] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.113681ms
[2022-09-10 17:48:08,924] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd2512b97-d142-4da0-b739-d8114b074438', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '45.113681ms', 'executionTime': '45.05275ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:48:08,951] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9f623c04a9b94953b77a8608de842b21join_yr'
[2022-09-10 17:48:08,977] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9f623c04a9b94953b77a8608de842b21join_yr%27
[2022-09-10 17:48:08,980] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.17833ms
[2022-09-10 17:48:08,980] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0ff9fdab-f223-47a6-ba44-e5647f24e3f6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.17833ms', 'executionTime': '1.108939ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:48:09,032] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:48:09,162] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:48:09,165] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:09,165] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:09,881] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:09,941] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:48:09,941] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:48:10,061] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:48:10,116] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:48:10,119] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:10,119] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:10,841] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:10,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:48:10,896] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:48:11,018] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:48:11,077] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:48:11,077] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:48:11,078] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:48:11,103] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:48:11,128] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:48:11,138] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.785566ms
[2022-09-10 17:48:11,138] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b0795bdc-5c24-46cc-af1b-fe0c941c26e5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.785566ms', 'executionTime': '8.717822ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:48:11,139] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:48:11,164] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:48:11,189] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:48:11,192] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.03184ms
[2022-09-10 17:48:11,192] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd3219fac-7c59-4d7f-a36e-cf0b6a3394ae', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.03184ms', 'executionTime': '969.247┬Ás', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:48:11,243] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:48:11,243] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.02909324472283, 'mem_free': 12949233664, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:48:11,243] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:48:11,246] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:11,247] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:11,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:11,993] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:11,994] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:12,867] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:12,875] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:12,876] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:14,156] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:14,167] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:14,167] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:15,402] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:23,096] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 17:48:23,238] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:48:23,553] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:48:23,581] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:48:23,581] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:48:23,637] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:48:23,691] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:48:23,744] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:48:23,744] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:48:23,822] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:48:23,823] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:48:23,848] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:23,978] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:48:23,978] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:24,005] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:48:24,031] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:48:24,031] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:24,058] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:48:24,083] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:48:24,083] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:24,109] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:48:24,135] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:48:24,135] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:24,161] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:48:24,161] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 17:48:24,162] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:48:24,162] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 97.447s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_17-31-04/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-10 17:48:24,219] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:24,219] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:24,916] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:24,949] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:48:25,028] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 17:48:25,029] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-10 17:48:25,029] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 17:48:25,244] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 17:48:25,271] - [task:165] INFO -  {'uptime': '1036', 'memoryTotal': 15466930176, 'memoryFree': 12952780800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:48:25,298] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 17:48:25,298] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 17:48:25,299] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 17:48:25,334] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 17:48:25,368] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 17:48:25,368] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 17:48:25,396] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 17:48:25,396] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 17:48:25,397] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 17:48:25,397] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 17:48:25,444] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:48:25,447] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:25,447] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:26,161] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:26,163] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:48:26,302] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:48:26,303] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:48:26,334] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:26,361] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:26,390] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:48:26,521] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 17:48:26,549] - [task:165] INFO -  {'uptime': '1034', 'memoryTotal': 15466930176, 'memoryFree': 12952604672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:48:26,575] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:48:26,604] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 17:48:26,605] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 17:48:26,658] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:48:26,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:26,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:27,403] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:27,404] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:48:27,544] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:48:27,545] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:48:27,576] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:27,602] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:27,631] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:48:27,754] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 17:48:27,781] - [task:165] INFO -  {'uptime': '1035', 'memoryTotal': 15466930176, 'memoryFree': 12951343104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:48:27,808] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:48:27,836] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 17:48:27,837] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 17:48:27,888] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:48:27,891] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:27,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:28,612] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:28,613] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:48:28,742] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:48:28,743] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:48:28,772] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:28,798] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:28,827] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:48:28,943] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 17:48:28,970] - [task:165] INFO -  {'uptime': '1035', 'memoryTotal': 15466930176, 'memoryFree': 12943654912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 17:48:28,996] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 17:48:29,023] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 17:48:29,024] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 17:48:29,073] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 17:48:29,076] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:29,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:29,758] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:29,759] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:48:29,891] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:48:29,892] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:48:29,921] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:29,947] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 17:48:29,976] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 17:48:30,074] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 17:48:30,627] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:48:35,633] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 17:48:35,719] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 17:48:35,724] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:35,724] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:36,434] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:36,435] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 17:48:36,561] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 17:48:36,562] - [remote_util:5249] INFO - b'ok'
[2022-09-10 17:48:36,562] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 17:48:37,236] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 17:48:37,293] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 17:48:37,293] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 17:48:38,208] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:48:38,513] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:48:38,795] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 17:48:38,799] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-10 17:48:38,859] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:38,859] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:39,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:39,553] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:39,553] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:40,608] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:40,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:40,616] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:41,874] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:41,886] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:41,886] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:43,109] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:50,528] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:48:50,528] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.58247530315155, 'mem_free': 12948787200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:48:50,528] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:48:50,529] - [basetestcase:467] INFO - Time to execute basesetup : 26.31272602081299
[2022-09-10 17:48:50,583] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:48:50,583] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:48:50,637] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:48:50,637] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:48:50,692] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:48:50,692] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:48:50,748] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 17:48:50,748] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 17:48:50,804] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:48:50,868] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 17:48:50,868] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 17:48:50,869] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 17:48:55,880] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 17:48:55,884] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:48:55,884] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:48:56,614] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:48:57,814] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 17:48:57,992] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 17:49:00,923] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 17:49:01,000] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:49:01,000] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 17:49:01,001] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 17:49:31,026] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 17:49:31,055] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:49:31,081] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:49:31,147] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.676826ms
[2022-09-10 17:49:31,147] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2bfd8b1b-29b0-41d1-9eef-131e64b9dfb4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.676826ms', 'executionTime': '64.607025ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 17:49:31,148] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 17:49:31,173] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 17:49:31,198] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 17:49:31,948] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 748.070692ms
[2022-09-10 17:49:31,948] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 17:49:32,004] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:49:32,034] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:49:32,042] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.463386ms
[2022-09-10 17:49:32,253] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:49:32,290] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 17:49:32,319] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 17:49:32,320] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 17:49:32,331] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 17:49:32,390] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:49:32,524] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:49:32,527] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:49:32,528] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:49:33,314] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:49:33,369] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:49:33,370] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 17:49:33,491] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:49:33,548] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:49:33,552] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:49:33,552] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:49:34,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:49:34,387] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 17:49:34,387] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 17:49:34,511] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 17:49:34,568] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 17:49:34,568] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 17:49:34,568] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:49:34,594] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 17:49:34,620] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 17:49:34,630] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.650032ms
[2022-09-10 17:49:34,655] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 17:49:34,681] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 17:49:34,728] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.973189ms
[2022-09-10 17:49:34,808] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 17:49:34,808] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.986919885234606, 'mem_free': 12779520000, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 17:49:34,809] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 17:49:34,813] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:49:34,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:49:35,558] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:49:35,562] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:49:35,563] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:49:36,829] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:49:36,842] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:49:36,842] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:49:38,146] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:49:38,158] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 17:49:38,158] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 17:49:39,482] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 17:49:47,090] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-10 17:49:47,241] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 17:49:47,767] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 17:49:47,795] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 17:49:47,796] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 17:49:47,849] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:49:47,901] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:49:47,954] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 17:49:47,955] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 17:49:48,035] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 17:49:48,036] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 17:49:48,062] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:49:48,190] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 17:49:48,190] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:49:48,217] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 17:49:48,242] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 17:49:48,242] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:49:48,268] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 17:49:48,294] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 17:49:48,294] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:49:48,320] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 17:49:48,345] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 17:49:48,345] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 17:49:48,380] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 17:49:48,381] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-10 17:49:48,381] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 17:49:48,382] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 84.243s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 21612 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-10.09.2022-16.00.cfg
Build Log: make-10.09.2022-16.00.log
Server Log: logs-10.09.2022-16.00.tar.gz

Finished