Building

Started building at 2022/09/11 04:09:08
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 11.09.2022-04.09
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at 0760e59, changes since last good build: 
fatal: Invalid revision range 67d5c96bc6a9c01fadbcc5dbdd2a62a39593b413..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable 48ceb1cc] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable ae8a18d4] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 2c982d11] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable 03aa63fe] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable 2f8308eb] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable aa67da54] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable e0e4cecd] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable b519cc81] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable d09d7603] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 53ecd768] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable c789a714] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 8578e1aa] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable f150bd79] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 10871dd] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable 3000d0f] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable cc26a77] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 4c75296] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 3b1c3cd] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable 8a1fb1d] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable feaa6a2] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 52396bd] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/11 05:03:32
Testing mode: sanity,unit,functional,integration
Using storage type: plasma
Setting ulimit to 200000

Simple Test

Sep 11 05:08:36 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 05:12:28 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 05:13:13 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 11 05:14:43 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 11 05:15:37 b" 'stop_on_failure': 'True'}"
Sep 11 05:15:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 05:15:37 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_1'}"
Sep 11 05:15:37 b'-->result: '
Sep 11 05:15:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 11 05:15:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 05:15:37 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_2'}"
Sep 11 05:15:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 05:15:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 11 05:15:37 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_3'}"
Sep 11 05:15:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 05:15:37 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 05:15:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 11 05:15:37 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_4'}"
Sep 11 05:15:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 05:15:37 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 05:15:37 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 05:25:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 11 05:26:11 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 11 05:29:57 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 05:33:52 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 05:36:37 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 05:36:37 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_5'}"
Sep 11 05:36:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 05:36:37 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 11 05:36:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 11 05:36:37 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_6'}"
Sep 11 05:36:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 05:36:37 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 05:36:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 11 05:36:37 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_7'}"
Sep 11 05:36:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 05:36:37 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 05:36:37 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 11 05:36:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 11 05:36:37 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-03-53/test_8'}"
Sep 11 05:36:37 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 05:36:37 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 05:36:37 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 05:36:37 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 11 05:36:37 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 11 05:36:39 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 05:36:39 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 05:36:39 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 11 05:36:39 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 11 05:36:39 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 11 05:36:39 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 11 05:36:39 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 11 05:36:39 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 187: go: command not found
/home/buildbot/bin/dotest: line 190: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=plasma
Printing gsi_type=plasma
gsi_type=plasma
In here
-p makefile=True,gsi_type=plasma
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'plasma',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-11 05:39:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:39:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:39:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:39:31 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 05:39:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 05:39:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:39:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:39:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:39:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:39:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:39:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:39:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:39:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:39:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:39:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:39:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:39:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:39:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 05:40:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:40:00 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-11 05:40:01 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 2071379968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:40:01 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:40:02 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1900761088, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:40:02 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 05:40:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:40:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1765781504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:40:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:40:04 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1681182720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:40:04 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:40:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:40:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:40:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:40:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:40:05 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 05:40:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 05:40:05 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-11 05:40:05 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 05:40:10 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 05:40:10 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-11 05:40:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:40:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:40:11 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:40:11 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 05:40:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 05:40:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-11 05:40:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 05:40:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 05:40:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 05:40:13 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 66.1148710041576, 'mem_free': 13390516224, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.942453384399414
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 05:40:16 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 05:40:21 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 05:40:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:23 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 05:40:23 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 05:40:24 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 05:40:25 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:40:25 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 05:40:25 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 05:40:55 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 05:40:55 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 05:40:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 05:40:55 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 127.056562ms
2022-09-11 05:40:55 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '76cffd2e-82c7-4d03-9a6e-845f4bd68e33', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '127.056562ms', 'executionTime': '126.865831ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 05:40:55 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 05:40:55 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 05:40:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 792.867353ms
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.847266ms
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 05:40:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.229478ms
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 64.312077ms
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.694564248230244, 'mem_free': 13372923904, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:40:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:40:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:40:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 05:41:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 05:41:03 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 05:41:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 92.251s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-11 05:41:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.78310474300358, 'mem_free': 13322665984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 05:41:09 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 05:41:09 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-11 05:41:10 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 05:41:10 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13341347840, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:41:10 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 05:41:10 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 05:41:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 05:41:11 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13346152448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13347164160, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13342662656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 05:41:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 05:41:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-11 05:41:13 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 05:41:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 05:41:19 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 05:41:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 05:41:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-11 05:41:20 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 05:41:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 05:41:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 05:41:22 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.91664453915519, 'mem_free': 13331062784, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 22.43454647064209
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 05:41:26 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 05:41:31 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 05:41:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:41:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:41:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:41:32 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 05:41:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 05:41:34 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 05:41:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:41:34 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 05:41:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 05:42:04 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 05:42:04 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 05:42:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 05:42:04 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 65.213827ms
2022-09-11 05:42:04 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '1fd3db92-8fe1-4eb3-89e2-f868a90648cd', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.213827ms', 'executionTime': '65.096563ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 05:42:04 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 05:42:04 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 05:42:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 05:42:05 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 714.327301ms
2022-09-11 05:42:05 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 05:42:05 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 05:42:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 05:42:05 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.878966ms
2022-09-11 05:42:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:42:06 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 05:42:06 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 05:42:06 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 05:42:06 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 05:42:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:42:07 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeec9db2776aba54e4087013cc9a0e43530job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-11 05:42:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeec9db2776aba54e4087013cc9a0e43530job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-11 05:42:07 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 51.399086ms
2022-09-11 05:42:07 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeec9db2776aba54e4087013cc9a0e43530job_title) USING GSI
2022-09-11 05:42:08 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeec9db2776aba54e4087013cc9a0e43530job_title) USING GSI
2022-09-11 05:42:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeec9db2776aba54e4087013cc9a0e43530job_title%29+USING+GSI
2022-09-11 05:42:08 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 23.94198ms
2022-09-11 05:42:09 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeec9db2776aba54e4087013cc9a0e43530job_title'
2022-09-11 05:42:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec9db2776aba54e4087013cc9a0e43530job_title%27
2022-09-11 05:42:09 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.122616ms
2022-09-11 05:42:10 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeec9db2776aba54e4087013cc9a0e43530job_title'
2022-09-11 05:42:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec9db2776aba54e4087013cc9a0e43530job_title%27
2022-09-11 05:42:10 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.913369ms
2022-09-11 05:42:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 05:42:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 05:42:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-11 05:42:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.988314ms
2022-09-11 05:42:11 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '80918662-61d1-4dde-a65f-c5955ce51a7d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec9db2776aba54e4087013cc9a0e43530job_title', 'index_id': 'ee256aaabaa6118', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.988314ms', 'executionTime': '2.871327ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
2022-09-11 05:42:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 05:42:11 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 05:42:11 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 05:42:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-11 05:42:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-11 05:42:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-11 05:42:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-11 05:42:11 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-11 05:42:11 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-11 05:42:12 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 05:42:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 05:42:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-11 05:42:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 144.515009ms
2022-09-11 05:42:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-11 05:42:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-11 05:42:13 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 05:42:13 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeec9db2776aba54e4087013cc9a0e43530job_title'
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec9db2776aba54e4087013cc9a0e43530job_title%27
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.957898ms
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeec9db2776aba54e4087013cc9a0e43530job_title ON default USING GSI
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeec9db2776aba54e4087013cc9a0e43530job_title+ON+default+USING+GSI
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 42.894234ms
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeec9db2776aba54e4087013cc9a0e43530job_title'
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec9db2776aba54e4087013cc9a0e43530job_title%27
2022-09-11 05:42:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.545255ms
2022-09-11 05:42:14 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '78e9aa7b-19f0-42f7-91f3-fc3b31c86269', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.545255ms', 'executionTime': '5.450127ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 05:42:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:42:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:42:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:42:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.403376ms
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 55.329903ms
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.81495606779957, 'mem_free': 13231325184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:42:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 05:42:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 05:42:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 05:42:20 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 05:42:20 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 05:42:21 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 05:42:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.998s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_2'}
[2022-09-11 05:42:21,920] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:21,920] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:22,218] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:22,249] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:42:22,330] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:42:22,330] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-11 05:42:22,331] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:42:22,950] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:42:22,978] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13259513856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:42:23,007] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:42:23,007] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:42:23,008] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:42:23,039] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:42:23,075] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:42:23,075] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:42:23,105] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:42:23,105] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:42:23,106] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:42:23,106] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:42:23,156] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:42:23,159] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:23,159] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:23,459] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:23,460] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:42:23,535] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:42:23,536] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:42:23,567] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:23,595] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:23,626] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:42:23,756] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:42:23,785] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13226004480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:42:23,813] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:42:23,842] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:42:23,842] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:42:23,894] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:42:23,897] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:23,897] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:24,185] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:24,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:42:24,262] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:42:24,263] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:42:24,292] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:24,319] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:24,347] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:42:24,470] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:42:24,499] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13257732096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:42:24,526] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:42:24,554] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:42:24,554] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:42:24,605] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:42:24,610] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:24,610] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:24,873] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:24,875] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:42:24,941] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:42:24,942] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:42:24,971] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:24,997] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:25,026] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:42:25,145] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:42:25,171] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13225127936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:42:25,197] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:42:25,226] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:42:25,226] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:42:25,276] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:42:25,279] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:25,279] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:25,540] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:25,541] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:42:25,608] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:42:25,609] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:42:25,639] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:25,665] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:42:25,695] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:42:25,790] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:42:26,339] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:42:31,345] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:42:31,426] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:42:31,430] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:31,430] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:31,704] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:31,705] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:42:31,773] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:42:31,774] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:42:31,775] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:42:32,904] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:42:32,959] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 05:42:32,959] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:42:33,956] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:42:34,255] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:42:34,572] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:42:34,574] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:42:34,635] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:34,635] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:34,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:34,926] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:34,926] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:35,195] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:35,201] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:35,201] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:35,451] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:35,456] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:35,456] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:35,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:39,339] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:42:39,340] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 30.47205083377891, 'mem_free': 13255352320, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:42:39,340] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:42:39,340] - [basetestcase:467] INFO - Time to execute basesetup : 17.42258596420288
[2022-09-11 05:42:39,391] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:42:39,391] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:42:39,446] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:42:39,447] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:42:39,501] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:42:39,501] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:42:39,553] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:42:39,553] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:42:39,606] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:42:39,672] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:42:39,673] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:42:39,673] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:42:44,683] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:42:44,686] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:42:44,686] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:42:44,947] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:42:45,888] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:42:46,053] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:42:48,717] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:42:48,739] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 05:42:49,380] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:42:49,380] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:42:49,380] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:43:19,402] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:43:19,431] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:43:19,455] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:43:19,523] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.960607ms
[2022-09-11 05:43:19,523] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '15aba109-a829-4172-81f7-be49494af27e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.960607ms', 'executionTime': '65.897218ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:43:19,523] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:43:19,548] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:43:19,573] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:43:20,250] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 675.67813ms
[2022-09-11 05:43:20,251] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:43:20,318] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:43:20,357] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:43:20,365] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.088254ms
[2022-09-11 05:43:20,558] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:43:20,600] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:43:20,614] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:43:20,615] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:43:20,628] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:43:20,701] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:43:21,526] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3b4eb43124bd47d5a1cf7762db5771f2job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:43:21,554] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3b4eb43124bd47d5a1cf7762db5771f2job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:43:21,605] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.462039ms
[2022-09-11 05:43:21,606] - [base_gsi:282] INFO - BUILD INDEX on default(employee3b4eb43124bd47d5a1cf7762db5771f2job_title) USING GSI
[2022-09-11 05:43:22,635] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee3b4eb43124bd47d5a1cf7762db5771f2job_title) USING GSI
[2022-09-11 05:43:22,661] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee3b4eb43124bd47d5a1cf7762db5771f2job_title%29+USING+GSI
[2022-09-11 05:43:22,689] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.547228ms
[2022-09-11 05:43:23,720] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b4eb43124bd47d5a1cf7762db5771f2job_title'
[2022-09-11 05:43:23,746] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b4eb43124bd47d5a1cf7762db5771f2job_title%27
[2022-09-11 05:43:23,755] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.577427ms
[2022-09-11 05:43:24,786] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b4eb43124bd47d5a1cf7762db5771f2job_title'
[2022-09-11 05:43:24,811] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b4eb43124bd47d5a1cf7762db5771f2job_title%27
[2022-09-11 05:43:24,821] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.342594ms
[2022-09-11 05:43:25,270] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 05:43:25,435] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:43:26,388] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:43:26,824] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:43:26,857] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:43:26,882] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 05:43:26,886] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.354299ms
[2022-09-11 05:43:26,886] - [task:3246] INFO - {'requestID': '60c29a06-88e8-4727-9c2c-5a97875c0022', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee3b4eb43124bd47d5a1cf7762db5771f2job_title', 'index_id': '759b46ee24207bd0', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.354299ms', 'executionTime': '2.285305ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 05:43:26,887] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:43:26,887] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:43:26,887] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:43:26,888] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:43:26,888] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 05:43:26,888] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:43:26,889] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:43:26,889] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:43:26,889] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:43:27,888] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:43:27,920] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:43:27,947] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 05:43:28,110] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 154.206861ms
[2022-09-11 05:43:28,111] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:43:28,111] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:43:28,885] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:43:28,885] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:43:29,917] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b4eb43124bd47d5a1cf7762db5771f2job_title'
[2022-09-11 05:43:29,943] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b4eb43124bd47d5a1cf7762db5771f2job_title%27
[2022-09-11 05:43:29,954] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.212194ms
[2022-09-11 05:43:29,979] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee3b4eb43124bd47d5a1cf7762db5771f2job_title ON default USING GSI
[2022-09-11 05:43:30,005] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee3b4eb43124bd47d5a1cf7762db5771f2job_title+ON+default+USING+GSI
[2022-09-11 05:43:30,073] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.283114ms
[2022-09-11 05:43:30,106] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b4eb43124bd47d5a1cf7762db5771f2job_title'
[2022-09-11 05:43:30,132] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b4eb43124bd47d5a1cf7762db5771f2job_title%27
[2022-09-11 05:43:30,139] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.187707ms
[2022-09-11 05:43:30,140] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c31c22a1-ce56-49c1-aea7-5786f3ebad9c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.187707ms', 'executionTime': '6.132742ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:43:30,246] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:43:30,249] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:30,249] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:30,585] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:30,640] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:43:30,640] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:43:30,702] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:43:30,763] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:43:30,766] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:30,766] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:31,133] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:31,195] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:43:31,195] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:43:31,262] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:43:31,320] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:43:31,320] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:43:31,320] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:43:31,346] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:43:31,373] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:43:31,383] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.854018ms
[2022-09-11 05:43:31,409] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:43:31,436] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:43:31,484] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.782735ms
[2022-09-11 05:43:31,554] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:43:31,555] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.62245593151238, 'mem_free': 13139603456, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:43:31,555] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:43:31,559] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:31,559] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:31,944] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:31,949] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:31,949] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:32,350] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:32,356] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:32,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:32,982] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:32,990] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:32,990] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:33,643] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:37,691] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:43:37,833] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:43:38,469] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:43:38,497] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:43:38,497] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:43:38,551] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:43:38,609] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:43:38,663] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:43:38,665] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:43:38,745] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:43:38,746] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:43:38,771] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:43:38,901] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:43:38,901] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:43:38,927] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:43:38,960] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:43:38,961] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:43:38,992] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:43:39,018] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:43:39,018] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:43:39,044] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:43:39,069] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:43:39,069] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:43:39,095] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:43:39,096] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:43:39,096] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:43:39,096] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 77.230s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_3'}
[2022-09-11 05:43:39,195] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:39,195] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:39,498] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:39,528] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:43:39,607] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:43:39,607] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-11 05:43:39,608] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:43:40,223] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:43:40,251] - [task:165] INFO -  {'uptime': '244', 'memoryTotal': 15466930176, 'memoryFree': 13189566464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:43:40,278] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:43:40,278] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:43:40,278] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:43:40,310] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:43:40,349] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:43:40,349] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:43:40,377] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:43:40,378] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:43:40,378] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:43:40,379] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:43:40,427] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:43:40,430] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:40,430] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:40,732] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:40,733] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:43:40,808] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:43:40,809] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:43:40,838] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:40,863] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:40,892] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:43:41,013] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:43:41,040] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13190737920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:43:41,065] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:43:41,093] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:43:41,094] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:43:41,148] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:43:41,152] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:41,152] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:41,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:41,462] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:43:41,536] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:43:41,537] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:43:41,567] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:41,592] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:41,621] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:43:41,737] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:43:41,763] - [task:165] INFO -  {'uptime': '240', 'memoryTotal': 15466930176, 'memoryFree': 13189984256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:43:41,788] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:43:41,817] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:43:41,817] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:43:41,870] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:43:41,874] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:41,874] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:42,200] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:42,202] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:43:42,278] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:43:42,280] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:43:42,311] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:42,339] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:42,368] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:43:42,491] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:43:42,519] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13191204864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:43:42,544] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:43:42,573] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:43:42,573] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:43:42,625] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:43:42,628] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:42,629] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:42,970] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:42,971] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:43:43,056] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:43:43,057] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:43:43,087] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:43,114] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:43:43,143] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:43:43,233] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:43:43,869] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:43:48,874] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:43:48,957] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:43:48,961] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:48,961] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:49,291] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:49,292] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:43:49,369] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:43:49,371] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:43:49,371] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:43:50,348] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:43:50,405] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 05:43:50,406] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:43:51,308] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:43:51,623] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:43:51,891] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:43:51,893] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:43:51,950] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:51,950] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:52,267] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:52,272] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:52,272] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:52,596] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:52,601] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:52,602] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:52,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:52,982] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:43:52,983] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:43:53,544] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:43:57,574] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:43:57,574] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.34181859530986, 'mem_free': 13194383360, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:43:57,575] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:43:57,575] - [basetestcase:467] INFO - Time to execute basesetup : 18.382379293441772
[2022-09-11 05:43:57,626] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:43:57,626] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:43:57,682] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:43:57,682] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:43:57,737] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:43:57,737] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:43:57,790] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:43:57,791] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:43:57,841] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:43:57,898] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:43:57,898] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:43:57,898] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:44:02,908] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:44:02,912] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:02,912] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:03,227] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:04,177] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:44:04,346] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:44:06,973] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:44:06,989] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 05:44:07,830] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:44:07,830] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:44:07,830] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:44:37,859] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:44:37,889] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:44:37,914] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:44:37,979] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.302754ms
[2022-09-11 05:44:37,980] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b4e35dbd-d9a3-46ab-9a66-ed820d1a21ea', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.302754ms', 'executionTime': '63.240741ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:44:37,980] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:44:38,005] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:44:38,031] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:44:38,742] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 708.63186ms
[2022-09-11 05:44:38,742] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:44:38,818] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:44:38,865] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:44:38,876] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.701398ms
[2022-09-11 05:44:39,090] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:44:39,125] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:44:39,142] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:44:39,142] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:44:39,159] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:44:39,227] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:44:40,051] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee432d171e645449fc9a5dbe07599b1c7ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:44:40,076] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee432d171e645449fc9a5dbe07599b1c7ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:44:40,129] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.984738ms
[2022-09-11 05:44:40,130] - [base_gsi:282] INFO - BUILD INDEX on default(employee432d171e645449fc9a5dbe07599b1c7ajob_title) USING GSI
[2022-09-11 05:44:41,161] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee432d171e645449fc9a5dbe07599b1c7ajob_title) USING GSI
[2022-09-11 05:44:41,188] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee432d171e645449fc9a5dbe07599b1c7ajob_title%29+USING+GSI
[2022-09-11 05:44:41,210] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.515012ms
[2022-09-11 05:44:42,241] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee432d171e645449fc9a5dbe07599b1c7ajob_title'
[2022-09-11 05:44:42,267] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee432d171e645449fc9a5dbe07599b1c7ajob_title%27
[2022-09-11 05:44:42,275] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.934435ms
[2022-09-11 05:44:43,304] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee432d171e645449fc9a5dbe07599b1c7ajob_title'
[2022-09-11 05:44:43,329] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee432d171e645449fc9a5dbe07599b1c7ajob_title%27
[2022-09-11 05:44:43,336] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.479004ms
[2022-09-11 05:44:43,705] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 05:44:43,962] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:44:44,636] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:44:45,339] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:44:45,373] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:44:45,399] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 05:44:45,403] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.26826ms
[2022-09-11 05:44:45,403] - [task:3246] INFO - {'requestID': '586f1f4c-5bea-4fdf-a14b-bb054d4e656f', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee432d171e645449fc9a5dbe07599b1c7ajob_title', 'index_id': '7e522fee4175b9b1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.26826ms', 'executionTime': '2.196427ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 05:44:45,404] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:44:45,404] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:44:45,404] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:44:45,405] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:44:45,405] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 05:44:45,405] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:44:45,406] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:44:45,406] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:44:45,406] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:44:46,405] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:44:46,436] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:44:46,461] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 05:44:46,592] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 121.139693ms
[2022-09-11 05:44:46,592] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:44:46,593] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:44:47,373] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:44:47,374] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:44:48,406] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee432d171e645449fc9a5dbe07599b1c7ajob_title'
[2022-09-11 05:44:48,431] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee432d171e645449fc9a5dbe07599b1c7ajob_title%27
[2022-09-11 05:44:48,442] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.928861ms
[2022-09-11 05:44:48,469] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee432d171e645449fc9a5dbe07599b1c7ajob_title ON default USING GSI
[2022-09-11 05:44:48,495] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee432d171e645449fc9a5dbe07599b1c7ajob_title+ON+default+USING+GSI
[2022-09-11 05:44:48,531] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.372948ms
[2022-09-11 05:44:48,568] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee432d171e645449fc9a5dbe07599b1c7ajob_title'
[2022-09-11 05:44:48,596] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee432d171e645449fc9a5dbe07599b1c7ajob_title%27
[2022-09-11 05:44:48,603] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.91315ms
[2022-09-11 05:44:48,604] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a10f456e-92e6-47c8-9937-753e34d1d369', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.91315ms', 'executionTime': '5.853219ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:44:48,709] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:44:48,712] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:48,713] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:49,127] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:49,186] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:44:49,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:44:49,258] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:44:49,316] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:44:49,319] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:49,319] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:49,753] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:49,813] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:44:49,814] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:44:49,888] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:44:49,945] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:44:49,945] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:44:49,945] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:44:49,971] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:44:49,997] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:44:50,006] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.581268ms
[2022-09-11 05:44:50,034] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:44:50,061] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:44:50,133] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.397269ms
[2022-09-11 05:44:50,202] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:44:50,202] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.87163141955894, 'mem_free': 13094064128, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:44:50,202] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:44:50,206] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:50,206] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:50,605] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:50,610] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:50,610] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:51,227] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:51,234] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:51,234] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:51,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:51,978] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:51,978] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:52,669] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:56,960] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:44:57,104] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:44:57,915] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:44:57,944] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:44:57,944] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:44:58,003] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:44:58,058] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:44:58,111] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:44:58,112] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:44:58,191] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:44:58,192] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:44:58,227] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:44:58,365] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:44:58,366] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:44:58,394] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:44:58,421] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:44:58,421] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:44:58,449] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:44:58,477] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:44:58,477] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:44:58,504] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:44:58,530] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:44:58,531] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:44:58,559] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:44:58,559] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:44:58,559] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:44:58,560] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.422s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_4'}
[2022-09-11 05:44:58,665] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:58,666] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:44:59,080] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:44:59,112] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:44:59,199] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:44:59,200] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-11 05:44:59,201] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:44:59,692] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:44:59,720] - [task:165] INFO -  {'uptime': '324', 'memoryTotal': 15466930176, 'memoryFree': 13146861568, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:44:59,747] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:44:59,747] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:44:59,748] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:44:59,778] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:44:59,816] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:44:59,816] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:44:59,845] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:44:59,846] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:44:59,847] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:44:59,847] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:44:59,896] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:44:59,899] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:44:59,899] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:00,293] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:00,295] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:45:00,385] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:45:00,386] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:45:00,416] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:00,442] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:00,471] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:45:00,591] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:45:00,620] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13147385856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:45:00,646] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:45:00,675] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:45:00,675] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:45:00,731] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:45:00,734] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:00,734] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:01,110] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:01,111] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:45:01,193] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:45:01,194] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:45:01,226] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:01,252] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:01,281] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:45:01,403] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:45:01,431] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13148758016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:45:01,461] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:45:01,491] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:45:01,491] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:45:01,544] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:45:01,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:01,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:01,960] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:01,962] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:45:02,056] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:45:02,057] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:45:02,091] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:02,119] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:02,150] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:45:02,275] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:45:02,303] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13146419200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:45:02,330] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:45:02,360] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:45:02,361] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:45:02,415] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:45:02,417] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:02,418] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:02,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:02,840] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:45:02,943] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:45:02,944] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:45:02,975] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:03,002] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:45:03,033] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:45:03,125] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:45:03,660] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:45:08,662] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:45:08,747] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:45:08,752] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:08,752] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:09,154] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:09,155] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:45:09,240] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:45:09,241] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:45:09,242] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:45:10,251] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:45:10,311] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 05:45:10,312] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:45:11,205] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:45:11,507] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:45:11,762] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:45:11,765] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:45:11,818] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:11,818] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:12,212] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:12,217] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:12,217] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:12,601] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:12,606] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:12,606] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:13,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:13,204] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:13,205] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:13,907] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:18,309] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:45:18,309] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.82540514951456, 'mem_free': 13152522240, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:45:18,309] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:45:18,310] - [basetestcase:467] INFO - Time to execute basesetup : 19.647840976715088
[2022-09-11 05:45:18,362] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:45:18,362] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:45:18,417] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:45:18,417] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:45:18,471] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:45:18,472] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:45:18,526] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:45:18,526] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:45:18,577] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:45:18,639] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:45:18,640] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:45:18,641] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:45:23,654] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:45:23,657] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:45:23,657] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:45:24,023] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:45:24,976] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:45:25,218] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:45:27,759] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:45:27,776] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 05:45:28,464] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:45:28,464] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:45:28,464] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:45:58,492] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:45:58,522] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:45:58,547] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:45:58,614] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.729519ms
[2022-09-11 05:45:58,615] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a617403b-4634-4586-a46a-faf66616e273', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.729519ms', 'executionTime': '65.663449ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:45:58,615] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:45:58,641] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:45:58,669] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:45:59,378] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 706.679458ms
[2022-09-11 05:45:59,378] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:45:59,429] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:45:59,473] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:45:59,481] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.385342ms
[2022-09-11 05:45:59,678] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:45:59,714] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:45:59,728] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:45:59,728] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:45:59,743] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:45:59,823] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:46:00,648] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed28c44f548cc401c97cd33e37c38042ejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:46:00,674] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed28c44f548cc401c97cd33e37c38042ejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:46:00,724] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.641054ms
[2022-09-11 05:46:00,724] - [base_gsi:282] INFO - BUILD INDEX on default(employeed28c44f548cc401c97cd33e37c38042ejob_title) USING GSI
[2022-09-11 05:46:01,756] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed28c44f548cc401c97cd33e37c38042ejob_title) USING GSI
[2022-09-11 05:46:01,783] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed28c44f548cc401c97cd33e37c38042ejob_title%29+USING+GSI
[2022-09-11 05:46:01,810] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.961924ms
[2022-09-11 05:46:02,843] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed28c44f548cc401c97cd33e37c38042ejob_title'
[2022-09-11 05:46:02,868] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed28c44f548cc401c97cd33e37c38042ejob_title%27
[2022-09-11 05:46:02,875] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.624809ms
[2022-09-11 05:46:03,907] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed28c44f548cc401c97cd33e37c38042ejob_title'
[2022-09-11 05:46:03,934] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed28c44f548cc401c97cd33e37c38042ejob_title%27
[2022-09-11 05:46:03,942] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.766849ms
[2022-09-11 05:46:04,429] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 05:46:04,607] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:46:05,464] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:46:05,656] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-11 05:46:05,656] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:46:05,838] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-11 05:46:05,838] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-11 05:46:05,839] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2483475374, 0, b'')
[2022-09-11 05:46:05,945] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:46:05,975] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:46:06,004] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 05:46:06,008] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.244029ms
[2022-09-11 05:46:06,009] - [task:3246] INFO - {'requestID': '59444a9f-0eac-498b-b702-02091e5c0790', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeed28c44f548cc401c97cd33e37c38042ejob_title', 'index_id': '15bcb06a6e20b885', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.244029ms', 'executionTime': '2.175124ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 05:46:06,009] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:46:06,009] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:46:06,010] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:46:06,010] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:46:06,010] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 05:46:06,011] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:46:06,011] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:46:06,011] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:46:06,011] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:46:07,011] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:46:07,042] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:46:07,070] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 05:46:07,233] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 151.997751ms
[2022-09-11 05:46:07,233] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:46:07,233] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:46:07,994] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:46:07,995] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:46:09,027] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed28c44f548cc401c97cd33e37c38042ejob_title'
[2022-09-11 05:46:09,055] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed28c44f548cc401c97cd33e37c38042ejob_title%27
[2022-09-11 05:46:09,065] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.027635ms
[2022-09-11 05:46:09,092] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed28c44f548cc401c97cd33e37c38042ejob_title ON default USING GSI
[2022-09-11 05:46:09,118] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed28c44f548cc401c97cd33e37c38042ejob_title+ON+default+USING+GSI
[2022-09-11 05:46:09,159] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.793487ms
[2022-09-11 05:46:09,196] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed28c44f548cc401c97cd33e37c38042ejob_title'
[2022-09-11 05:46:09,225] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed28c44f548cc401c97cd33e37c38042ejob_title%27
[2022-09-11 05:46:09,233] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.725091ms
[2022-09-11 05:46:09,233] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '98c6c882-2c20-41a0-82fe-1f2a726167e9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.725091ms', 'executionTime': '5.599328ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:46:09,342] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:46:09,345] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:09,345] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:09,808] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:09,863] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:46:09,864] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:46:09,940] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:46:09,996] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:46:09,999] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:09,999] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:10,453] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:10,512] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:46:10,512] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:46:10,589] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:46:10,645] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:46:10,646] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:46:10,646] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:46:10,671] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:46:10,698] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:46:10,708] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.182765ms
[2022-09-11 05:46:10,735] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:46:10,760] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:46:10,813] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.943856ms
[2022-09-11 05:46:10,890] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:46:10,891] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.183530556894111, 'mem_free': 13044101120, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:46:10,891] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:46:10,895] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:10,895] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:11,332] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:11,337] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:11,337] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:11,938] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:11,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:11,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:12,716] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:12,723] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:12,724] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:13,489] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:18,138] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:46:18,281] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:46:18,796] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:46:18,824] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:46:18,824] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:46:18,879] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:46:18,932] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:46:18,984] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:46:18,985] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:46:19,063] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:46:19,064] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:46:19,089] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:46:19,217] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:46:19,218] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:46:19,249] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:46:19,277] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:46:19,277] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:46:19,305] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:46:19,331] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:46:19,331] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:46:19,358] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:46:19,383] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:46:19,384] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:46:19,420] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:46:19,421] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:46:19,421] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:46:19,421] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.812s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_5'}
[2022-09-11 05:46:19,523] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:19,524] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:19,984] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:20,017] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:46:20,095] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:46:20,096] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-11 05:46:20,097] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:46:20,551] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:46:20,578] - [task:165] INFO -  {'uptime': '405', 'memoryTotal': 15466930176, 'memoryFree': 13114834944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:46:20,604] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:46:20,604] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:46:20,605] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:46:20,641] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:46:20,673] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:46:20,674] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:46:20,702] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:46:20,703] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:46:20,703] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:46:20,703] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:46:20,752] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:46:20,755] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:20,755] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:21,152] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:21,153] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:46:21,244] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:46:21,245] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:46:21,276] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:21,303] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:21,332] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:46:21,451] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:46:21,477] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13116751872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:46:21,502] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:46:21,529] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:46:21,530] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:46:21,580] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:46:21,583] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:21,583] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:21,997] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:21,998] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:46:22,088] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:46:22,089] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:46:22,120] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:22,144] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:22,173] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:46:22,287] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:46:22,314] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13116649472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:46:22,339] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:46:22,365] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:46:22,366] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:46:22,417] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:46:22,421] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:22,422] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:22,835] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:22,836] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:46:22,924] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:46:22,925] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:46:22,954] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:22,979] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:23,009] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:46:23,125] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:46:23,151] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13115342848, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:46:23,177] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:46:23,206] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:46:23,207] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:46:23,257] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:46:23,260] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:23,260] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:23,662] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:23,664] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:46:23,751] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:46:23,752] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:46:23,781] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:23,809] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:46:23,838] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:46:23,922] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:46:24,457] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:46:29,462] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:46:29,547] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:46:29,550] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:29,551] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:29,972] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:29,973] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:46:30,066] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:46:30,067] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:46:30,068] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:46:31,069] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:46:31,126] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 05:46:31,127] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:46:32,047] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:46:32,368] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:46:32,649] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:46:32,652] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:46:32,720] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:32,720] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:33,195] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:33,200] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:33,200] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:33,606] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:33,611] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:33,611] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:34,325] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:34,337] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:34,337] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:35,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:39,831] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:46:39,831] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.62902540059523, 'mem_free': 13113643008, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:46:39,831] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:46:39,832] - [basetestcase:467] INFO - Time to execute basesetup : 20.310476303100586
[2022-09-11 05:46:39,884] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:46:39,884] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:46:39,944] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:46:39,944] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:46:40,000] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:46:40,000] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:46:40,054] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:46:40,054] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:46:40,105] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:46:40,169] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:46:40,169] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:46:40,171] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:46:45,184] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:46:45,187] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:46:45,187] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:46:45,589] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:46:46,565] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:46:46,812] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:46:49,282] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:46:49,299] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 05:46:50,360] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:46:50,360] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:46:50,360] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:47:20,390] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:47:20,420] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:47:20,444] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:47:20,512] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.471973ms
[2022-09-11 05:47:20,512] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c4f27ee8-d035-4de4-b5bf-f4d4d79fbeb5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.471973ms', 'executionTime': '65.402411ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:47:20,512] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:47:20,537] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:47:20,562] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:47:21,233] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 668.653727ms
[2022-09-11 05:47:21,233] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:47:21,302] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:47:21,347] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:47:21,355] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.072956ms
[2022-09-11 05:47:21,555] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:47:21,589] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:47:21,608] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:47:21,610] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:47:21,633] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:47:21,700] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:47:22,518] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea6774e9d3c924b18a8e02442ccd2a183job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:47:22,544] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeea6774e9d3c924b18a8e02442ccd2a183job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:47:22,615] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.348835ms
[2022-09-11 05:47:22,615] - [base_gsi:282] INFO - BUILD INDEX on default(employeea6774e9d3c924b18a8e02442ccd2a183job_title) USING GSI
[2022-09-11 05:47:23,646] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeea6774e9d3c924b18a8e02442ccd2a183job_title) USING GSI
[2022-09-11 05:47:23,671] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeea6774e9d3c924b18a8e02442ccd2a183job_title%29+USING+GSI
[2022-09-11 05:47:23,694] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.524539ms
[2022-09-11 05:47:24,726] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6774e9d3c924b18a8e02442ccd2a183job_title'
[2022-09-11 05:47:24,751] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6774e9d3c924b18a8e02442ccd2a183job_title%27
[2022-09-11 05:47:24,760] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.530099ms
[2022-09-11 05:47:25,791] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6774e9d3c924b18a8e02442ccd2a183job_title'
[2022-09-11 05:47:25,820] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6774e9d3c924b18a8e02442ccd2a183job_title%27
[2022-09-11 05:47:25,829] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.421011ms
[2022-09-11 05:47:26,305] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:47:26,476] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:47:30,150] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:47:30,834] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:47:30,867] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:47:30,892] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 05:47:30,897] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.141804ms
[2022-09-11 05:47:30,897] - [task:3246] INFO - {'requestID': 'c2a4447f-47b1-45f2-ac53-9dafdf9e8a98', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea6774e9d3c924b18a8e02442ccd2a183job_title', 'index_id': '5193f9e6cefbe668', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.141804ms', 'executionTime': '3.075892ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 05:47:30,898] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:47:30,898] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:47:30,898] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:47:30,899] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:47:30,899] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 05:47:30,899] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:47:30,899] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:47:30,900] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:47:30,900] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:47:31,899] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:47:31,928] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:47:31,953] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 05:47:32,143] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 175.927016ms
[2022-09-11 05:47:32,144] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:47:32,144] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:47:33,684] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:47:33,684] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:47:34,716] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6774e9d3c924b18a8e02442ccd2a183job_title'
[2022-09-11 05:47:34,741] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6774e9d3c924b18a8e02442ccd2a183job_title%27
[2022-09-11 05:47:34,752] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.138223ms
[2022-09-11 05:47:34,779] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea6774e9d3c924b18a8e02442ccd2a183job_title ON default USING GSI
[2022-09-11 05:47:34,805] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeea6774e9d3c924b18a8e02442ccd2a183job_title+ON+default+USING+GSI
[2022-09-11 05:47:34,863] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.400223ms
[2022-09-11 05:47:34,899] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6774e9d3c924b18a8e02442ccd2a183job_title'
[2022-09-11 05:47:34,927] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6774e9d3c924b18a8e02442ccd2a183job_title%27
[2022-09-11 05:47:34,934] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.080426ms
[2022-09-11 05:47:34,935] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ec002e2f-e3ba-467f-a391-6020784d7ef0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.080426ms', 'executionTime': '6.011049ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:47:35,048] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:47:35,052] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:35,053] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:35,616] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:35,671] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:47:35,671] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:47:35,762] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:47:35,819] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:47:35,822] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:35,822] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:36,346] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:36,404] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:47:36,405] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:47:36,495] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:47:36,550] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:47:36,550] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:47:36,550] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:47:36,575] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:47:36,601] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:47:36,611] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.659922ms
[2022-09-11 05:47:36,637] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:47:36,663] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:47:36,726] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.636094ms
[2022-09-11 05:47:36,785] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:47:36,785] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.85461268390454, 'mem_free': 12851658752, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:47:36,785] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:47:36,789] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:36,789] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:37,322] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:37,326] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:37,327] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:37,846] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:37,851] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:37,851] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:38,731] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:38,739] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:38,740] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:39,692] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:45,763] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:47:45,902] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:47:46,675] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:47:46,704] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:47:46,705] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:47:46,763] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:47:46,817] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:47:46,870] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:47:46,870] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:47:46,952] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:47:46,953] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:47:46,980] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:47:47,129] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:47:47,130] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:47:47,158] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:47:47,184] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:47:47,185] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:47:47,212] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:47:47,238] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:47:47,238] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:47:47,266] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:47:47,292] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:47:47,292] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:47:47,320] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:47:47,320] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:47:47,321] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:47:47,321] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 87.853s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_6'}
[2022-09-11 05:47:47,426] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:47,426] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:47,913] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:47,945] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:47:48,024] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:47:48,025] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-11 05:47:48,025] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:47:48,452] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:47:48,479] - [task:165] INFO -  {'uptime': '493', 'memoryTotal': 15466930176, 'memoryFree': 12923817984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:47:48,505] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:47:48,506] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:47:48,506] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:47:48,537] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:47:48,580] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:47:48,581] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:47:48,609] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:47:48,610] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:47:48,610] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:47:48,611] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:47:48,658] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:47:48,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:48,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:49,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:49,195] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:47:49,307] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:47:49,308] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:47:49,339] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:49,367] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:49,398] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:47:49,518] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:47:49,546] - [task:165] INFO -  {'uptime': '492', 'memoryTotal': 15466930176, 'memoryFree': 13039968256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:47:49,572] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:47:49,600] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:47:49,601] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:47:49,656] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:47:49,659] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:49,659] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:50,183] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:50,185] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:47:50,288] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:47:50,289] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:47:50,320] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:50,346] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:50,375] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:47:50,493] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:47:50,521] - [task:165] INFO -  {'uptime': '492', 'memoryTotal': 15466930176, 'memoryFree': 13040324608, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:47:50,546] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:47:50,573] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:47:50,574] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:47:50,623] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:47:50,627] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:50,627] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:51,142] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:51,143] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:47:51,253] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:47:51,254] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:47:51,285] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:51,312] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:51,342] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:47:51,463] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:47:51,492] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13039308800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:47:51,519] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:47:51,547] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:47:51,547] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:47:51,600] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:47:51,603] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:51,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:52,165] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:52,166] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:47:52,277] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:47:52,278] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:47:52,308] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:52,334] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:47:52,363] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:47:52,453] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:47:53,010] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:47:58,015] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:47:58,099] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:47:58,103] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:47:58,103] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:47:58,623] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:47:58,624] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:47:58,731] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:47:58,732] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:47:58,733] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:47:59,586] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:47:59,641] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 05:47:59,643] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:48:00,553] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:48:00,872] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:48:01,134] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:48:01,136] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:48:01,192] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:48:01,192] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:48:01,727] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:48:01,731] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:48:01,732] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:48:02,367] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:48:02,378] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:48:02,378] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:48:03,276] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:48:03,285] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:48:03,285] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:48:04,176] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:48:09,894] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:48:09,895] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.76085070726254, 'mem_free': 13044563968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:48:09,895] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:48:09,895] - [basetestcase:467] INFO - Time to execute basesetup : 22.471531867980957
[2022-09-11 05:48:09,945] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:48:09,945] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:48:09,997] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:48:09,998] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:48:10,051] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:48:10,052] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:48:10,109] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:48:10,110] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:48:10,160] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:48:10,227] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:48:10,227] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:48:10,227] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:48:15,240] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:48:15,243] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:48:15,243] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:48:15,795] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:48:16,801] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:48:17,063] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:48:19,280] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:48:19,298] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 05:48:20,688] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:48:20,688] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:48:20,689] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:48:50,708] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:48:50,739] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:48:50,766] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:48:50,835] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.935992ms
[2022-09-11 05:48:50,835] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6e024664-25ba-40e8-9ec9-558ec0e74e32', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.935992ms', 'executionTime': '66.873115ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:48:50,835] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:48:50,861] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:48:50,887] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:48:51,569] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 679.817297ms
[2022-09-11 05:48:51,569] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:48:51,618] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:48:51,658] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:48:51,666] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.559244ms
[2022-09-11 05:48:51,892] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:48:51,928] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:48:51,945] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:48:51,946] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:48:51,961] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:48:52,034] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:48:52,853] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeedf76532e592b4517831aad54ced1288ejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:48:52,881] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeedf76532e592b4517831aad54ced1288ejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:48:52,931] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.11238ms
[2022-09-11 05:48:52,932] - [base_gsi:282] INFO - BUILD INDEX on default(employeedf76532e592b4517831aad54ced1288ejob_title) USING GSI
[2022-09-11 05:48:53,965] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeedf76532e592b4517831aad54ced1288ejob_title) USING GSI
[2022-09-11 05:48:53,992] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeedf76532e592b4517831aad54ced1288ejob_title%29+USING+GSI
[2022-09-11 05:48:54,018] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.739403ms
[2022-09-11 05:48:55,050] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedf76532e592b4517831aad54ced1288ejob_title'
[2022-09-11 05:48:55,078] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedf76532e592b4517831aad54ced1288ejob_title%27
[2022-09-11 05:48:55,088] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.714774ms
[2022-09-11 05:48:56,120] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedf76532e592b4517831aad54ced1288ejob_title'
[2022-09-11 05:48:56,146] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedf76532e592b4517831aad54ced1288ejob_title%27
[2022-09-11 05:48:56,154] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.153371ms
[2022-09-11 05:48:56,651] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 05:48:56,815] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:48:57,953] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:48:58,564] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 05:48:58,750] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:49:00,052] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:49:00,456] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:49:00,635] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:49:04,311] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:49:05,174] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:49:05,207] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:49:05,234] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 05:49:05,239] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.046757ms
[2022-09-11 05:49:05,239] - [task:3246] INFO - {'requestID': '7d259439-183b-4708-adc4-96d78feec6c5', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeedf76532e592b4517831aad54ced1288ejob_title', 'index_id': '83fcd4f4fa46af76', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.046757ms', 'executionTime': '2.973553ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 05:49:05,239] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:49:05,239] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:49:05,240] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:49:05,240] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:49:05,241] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 05:49:05,241] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:49:05,241] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:49:05,241] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:49:05,241] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 05:49:06,241] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:49:06,270] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 05:49:06,295] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 05:49:06,693] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 176.578335ms
[2022-09-11 05:49:06,693] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:49:06,694] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:49:08,227] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:49:08,228] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 05:49:09,258] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedf76532e592b4517831aad54ced1288ejob_title'
[2022-09-11 05:49:09,283] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedf76532e592b4517831aad54ced1288ejob_title%27
[2022-09-11 05:49:09,292] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.019705ms
[2022-09-11 05:49:09,318] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeedf76532e592b4517831aad54ced1288ejob_title ON default USING GSI
[2022-09-11 05:49:09,343] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeedf76532e592b4517831aad54ced1288ejob_title+ON+default+USING+GSI
[2022-09-11 05:49:09,384] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.327211ms
[2022-09-11 05:49:09,420] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedf76532e592b4517831aad54ced1288ejob_title'
[2022-09-11 05:49:09,449] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedf76532e592b4517831aad54ced1288ejob_title%27
[2022-09-11 05:49:09,457] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.45041ms
[2022-09-11 05:49:09,457] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8c4586b5-b231-47a5-8f50-3090ceb1dc0a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.45041ms', 'executionTime': '6.376425ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:49:09,565] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:49:09,568] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:09,568] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:10,226] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:10,282] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:49:10,282] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:49:10,403] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:49:10,462] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:49:10,465] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:10,465] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:11,084] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:11,138] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:49:11,138] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:49:11,246] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:49:11,301] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:49:11,301] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:49:11,302] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:49:11,327] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:49:11,352] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:49:11,362] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.895859ms
[2022-09-11 05:49:11,388] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:49:11,414] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:49:11,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.584297ms
[2022-09-11 05:49:11,535] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:49:11,535] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.0500375382282, 'mem_free': 12819148800, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:49:11,535] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:49:11,539] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:11,539] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:12,182] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:12,187] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:12,187] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:12,828] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:12,832] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:12,832] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:13,765] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:13,773] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:13,773] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:14,900] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:22,569] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:49:22,720] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:49:23,179] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:49:23,208] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:49:23,208] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:49:23,261] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:49:23,314] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:49:23,367] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:49:23,368] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:49:23,446] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:49:23,447] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:49:23,471] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:49:23,597] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:49:23,598] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:49:23,624] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:49:23,650] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:49:23,650] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:49:23,676] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:49:23,701] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:49:23,701] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:49:23,727] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:49:23,753] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:49:23,753] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:49:23,780] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:49:23,780] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 05:49:23,781] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:49:23,781] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 96.415s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_7'}
[2022-09-11 05:49:23,933] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:23,934] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:24,572] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:24,606] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:49:24,688] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:49:24,689] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-11 05:49:24,689] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:49:24,959] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:49:24,987] - [task:165] INFO -  {'uptime': '589', 'memoryTotal': 15466930176, 'memoryFree': 12975034368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:49:25,013] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:49:25,014] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:49:25,014] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:49:25,053] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:49:25,092] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:49:25,092] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:49:25,122] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:49:25,123] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:49:25,123] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:49:25,123] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:49:25,173] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:49:25,176] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:25,176] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:25,817] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:25,818] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:49:25,941] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:49:25,942] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:49:25,971] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:25,997] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:26,026] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:49:26,155] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:49:26,181] - [task:165] INFO -  {'uptime': '588', 'memoryTotal': 15466930176, 'memoryFree': 12973187072, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:49:26,209] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:49:26,238] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:49:26,239] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:49:26,290] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:49:26,293] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:26,293] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:26,927] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:26,928] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:49:27,051] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:49:27,052] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:49:27,084] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:27,113] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:27,148] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:49:27,273] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:49:27,300] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12965666816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:49:27,327] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:49:27,355] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:49:27,356] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:49:27,408] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:49:27,413] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:27,413] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:28,035] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:28,036] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:49:28,151] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:49:28,152] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:49:28,183] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:28,217] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:28,247] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:49:28,367] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:49:28,395] - [task:165] INFO -  {'uptime': '588', 'memoryTotal': 15466930176, 'memoryFree': 12989628416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:49:28,423] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:49:28,453] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:49:28,453] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:49:28,505] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:49:28,509] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:28,509] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:29,157] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:29,158] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:49:29,283] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:49:29,284] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:49:29,315] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:29,340] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:49:29,370] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:49:29,462] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:49:29,979] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:49:34,985] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:49:35,072] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:49:35,077] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:35,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:35,666] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:35,667] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:49:35,781] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:49:35,782] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:49:35,782] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:49:36,596] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:49:36,649] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 05:49:36,650] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:49:37,529] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:49:37,830] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:49:38,082] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:49:38,086] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 05:49:38,139] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:38,139] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:38,755] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:38,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:38,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:39,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:39,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:39,647] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:40,718] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:40,727] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:40,727] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:41,822] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:48,290] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:49:48,291] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.73896296805997, 'mem_free': 12985700352, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:49:48,291] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:49:48,291] - [basetestcase:467] INFO - Time to execute basesetup : 24.360785961151123
[2022-09-11 05:49:48,344] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:49:48,345] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:49:48,399] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:49:48,399] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:49:48,454] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:49:48,454] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:49:48,510] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:49:48,510] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:49:48,563] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:49:48,622] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:49:48,622] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:49:48,622] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:49:53,631] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:49:53,635] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:49:53,636] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:49:54,229] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:49:55,280] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:49:55,460] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:49:57,696] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:49:57,785] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:49:57,785] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:49:57,785] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:50:27,813] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:50:27,842] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:50:27,867] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:50:27,937] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.470612ms
[2022-09-11 05:50:27,937] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '14e89fd4-6189-4d60-a866-2a5cbb95ba1d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.470612ms', 'executionTime': '67.412505ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:50:27,937] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:50:27,963] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:50:27,989] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:50:28,667] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 676.461835ms
[2022-09-11 05:50:28,668] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:50:28,729] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:50:28,759] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:50:28,767] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.324986ms
[2022-09-11 05:50:28,992] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:50:29,026] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:50:29,042] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:50:29,045] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:50:29,073] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:50:29,073] - [base_gsi:326] INFO - []
[2022-09-11 05:50:29,954] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec83d0a42ad844d18a8df1efe9531e1b3job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:50:29,981] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec83d0a42ad844d18a8df1efe9531e1b3job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:50:30,029] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.333977ms
[2022-09-11 05:50:30,064] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec83d0a42ad844d18a8df1efe9531e1b3join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:50:30,090] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec83d0a42ad844d18a8df1efe9531e1b3join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:50:30,163] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 71.09109ms
[2022-09-11 05:50:30,164] - [base_gsi:282] INFO - BUILD INDEX on default(employeec83d0a42ad844d18a8df1efe9531e1b3job_title,employeec83d0a42ad844d18a8df1efe9531e1b3join_yr) USING GSI
[2022-09-11 05:50:31,195] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec83d0a42ad844d18a8df1efe9531e1b3job_title,employeec83d0a42ad844d18a8df1efe9531e1b3join_yr) USING GSI
[2022-09-11 05:50:31,221] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec83d0a42ad844d18a8df1efe9531e1b3job_title%2Cemployeec83d0a42ad844d18a8df1efe9531e1b3join_yr%29+USING+GSI
[2022-09-11 05:50:31,252] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.66735ms
[2022-09-11 05:50:32,289] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec83d0a42ad844d18a8df1efe9531e1b3job_title'
[2022-09-11 05:50:32,315] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec83d0a42ad844d18a8df1efe9531e1b3job_title%27
[2022-09-11 05:50:32,323] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.993177ms
[2022-09-11 05:50:33,355] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec83d0a42ad844d18a8df1efe9531e1b3job_title'
[2022-09-11 05:50:33,382] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec83d0a42ad844d18a8df1efe9531e1b3job_title%27
[2022-09-11 05:50:33,389] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.861272ms
[2022-09-11 05:50:33,415] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec83d0a42ad844d18a8df1efe9531e1b3join_yr'
[2022-09-11 05:50:33,441] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec83d0a42ad844d18a8df1efe9531e1b3join_yr%27
[2022-09-11 05:50:33,444] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.512896ms
[2022-09-11 05:50:34,476] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec83d0a42ad844d18a8df1efe9531e1b3job_title'
[2022-09-11 05:50:34,501] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec83d0a42ad844d18a8df1efe9531e1b3job_title%27
[2022-09-11 05:50:34,511] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.825967ms
[2022-09-11 05:50:34,536] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec83d0a42ad844d18a8df1efe9531e1b3job_title ON default USING GSI
[2022-09-11 05:50:34,562] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec83d0a42ad844d18a8df1efe9531e1b3job_title+ON+default+USING+GSI
[2022-09-11 05:50:34,621] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.04935ms
[2022-09-11 05:50:34,654] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec83d0a42ad844d18a8df1efe9531e1b3join_yr'
[2022-09-11 05:50:34,681] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec83d0a42ad844d18a8df1efe9531e1b3join_yr%27
[2022-09-11 05:50:34,689] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.793678ms
[2022-09-11 05:50:34,716] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec83d0a42ad844d18a8df1efe9531e1b3join_yr ON default USING GSI
[2022-09-11 05:50:34,742] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec83d0a42ad844d18a8df1efe9531e1b3join_yr+ON+default+USING+GSI
[2022-09-11 05:50:34,780] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.869422ms
[2022-09-11 05:50:34,815] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec83d0a42ad844d18a8df1efe9531e1b3job_title'
[2022-09-11 05:50:34,841] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec83d0a42ad844d18a8df1efe9531e1b3job_title%27
[2022-09-11 05:50:34,849] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.896652ms
[2022-09-11 05:50:34,849] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '41099101-b615-402c-8ebf-6aea1a586a5f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.896652ms', 'executionTime': '5.838218ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:50:34,881] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec83d0a42ad844d18a8df1efe9531e1b3join_yr'
[2022-09-11 05:50:34,908] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec83d0a42ad844d18a8df1efe9531e1b3join_yr%27
[2022-09-11 05:50:34,911] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.456941ms
[2022-09-11 05:50:34,911] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0cec6561-3b95-4d63-b8e6-7a190851fba1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.456941ms', 'executionTime': '1.401743ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:50:35,013] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:50:35,018] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:35,019] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:35,621] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:35,677] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:50:35,678] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:50:35,774] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:50:35,833] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:50:35,835] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:35,836] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:36,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:36,473] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:50:36,473] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:50:36,580] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:50:36,636] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:50:36,636] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:50:36,636] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:50:36,662] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:50:36,691] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:50:36,701] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.676268ms
[2022-09-11 05:50:36,728] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:50:36,754] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:50:36,811] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.315526ms
[2022-09-11 05:50:36,882] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:50:36,882] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.623425325944805, 'mem_free': 12877627392, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:50:36,882] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:50:36,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:36,887] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:37,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:37,554] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:37,554] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:38,164] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:38,169] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:38,169] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:38,989] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:39,002] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:39,002] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:40,078] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:47,427] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-11 05:50:47,579] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:50:48,093] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:50:48,121] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:50:48,122] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:50:48,182] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:50:48,237] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:50:48,293] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:50:48,294] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:50:48,374] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:50:48,375] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:50:48,400] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:50:48,534] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:50:48,534] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:50:48,562] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:50:48,590] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:50:48,590] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:50:48,619] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:50:48,647] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:50:48,647] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:50:48,674] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:50:48,711] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:50:48,711] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:50:48,740] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:50:48,740] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 05:50:48,740] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:50:48,741] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.862s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_8'}
[2022-09-11 05:50:48,821] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:48,821] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:49,434] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:49,466] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:50:49,546] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:50:49,547] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-11 05:50:49,547] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:50:49,847] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:50:49,876] - [task:165] INFO -  {'uptime': '674', 'memoryTotal': 15466930176, 'memoryFree': 12996526080, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:50:49,902] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:50:49,902] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:50:49,902] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:50:49,937] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:50:49,969] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:50:49,969] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:50:49,998] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:50:49,998] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:50:49,999] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:50:49,999] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:50:50,046] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:50:50,049] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:50,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:50,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:50,691] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:50:50,814] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:50:50,815] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:50:50,845] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:50,873] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:50,903] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:50:51,024] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:50:51,050] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 12997058560, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:50:51,075] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:50:51,103] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:50:51,103] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:50:51,154] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:50:51,157] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:51,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:51,753] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:51,755] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:50:51,875] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:50:51,876] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:50:51,906] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:51,932] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:51,960] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:50:52,079] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:50:52,106] - [task:165] INFO -  {'uptime': '672', 'memoryTotal': 15466930176, 'memoryFree': 12996550656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:50:52,132] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:50:52,159] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:50:52,159] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:50:52,211] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:50:52,216] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:52,216] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:52,854] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:52,855] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:50:52,978] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:50:52,979] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:50:53,014] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:53,040] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:53,068] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:50:53,189] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:50:53,217] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 12996689920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:50:53,243] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:50:53,271] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:50:53,272] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:50:53,325] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:50:53,328] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:50:53,328] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:50:53,967] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:50:53,968] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:50:54,093] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:50:54,094] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:50:54,126] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:54,152] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:50:54,184] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:50:54,277] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:50:54,929] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:50:59,934] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:51:00,025] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:51:00,029] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:51:00,030] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:51:00,688] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:51:00,689] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:51:00,818] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:51:00,819] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:51:00,819] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:51:01,412] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:51:01,470] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 05:51:01,471] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:51:02,526] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:51:02,853] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:51:03,159] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:51:03,163] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 05:51:03,216] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:51:03,216] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:51:03,833] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:51:03,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:51:03,839] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:51:04,767] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:51:04,776] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:51:04,776] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:51:05,884] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:51:05,893] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:51:05,893] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:51:06,955] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:51:13,416] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:51:13,417] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.91246833028407, 'mem_free': 12999798784, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:51:13,417] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:51:13,417] - [basetestcase:467] INFO - Time to execute basesetup : 24.59900426864624
[2022-09-11 05:51:13,467] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:51:13,467] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:51:13,520] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:51:13,520] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:51:13,572] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:51:13,572] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:51:13,625] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:51:13,625] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:51:13,676] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:51:13,742] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:51:13,742] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:51:13,742] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:51:18,754] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:51:18,758] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:51:18,759] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:51:19,346] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:51:20,375] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:51:20,662] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:51:22,814] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:51:22,895] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:51:22,895] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:51:22,895] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:51:52,926] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:51:52,955] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:51:52,980] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:51:53,052] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.17226ms
[2022-09-11 05:51:53,052] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a344c434-c2ee-4f1f-acd9-6aabe40bf9f9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.17226ms', 'executionTime': '69.104078ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:51:53,052] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:51:53,079] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:51:53,104] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:51:53,867] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 761.448095ms
[2022-09-11 05:51:53,868] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:51:53,941] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:51:53,979] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:51:53,988] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.757426ms
[2022-09-11 05:51:54,202] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:51:54,241] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:51:54,262] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:51:54,264] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:51:54,293] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:51:54,296] - [base_gsi:326] INFO - []
[2022-09-11 05:51:55,163] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:51:55,191] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:51:55,251] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.429446ms
[2022-09-11 05:51:55,251] - [base_gsi:282] INFO - BUILD INDEX on default(employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr) USING GSI
[2022-09-11 05:51:56,282] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr) USING GSI
[2022-09-11 05:51:56,308] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr%29+USING+GSI
[2022-09-11 05:51:56,332] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.595232ms
[2022-09-11 05:51:57,363] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr'
[2022-09-11 05:51:57,389] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr%27
[2022-09-11 05:51:57,396] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.65727ms
[2022-09-11 05:51:58,429] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr'
[2022-09-11 05:51:58,459] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr%27
[2022-09-11 05:51:58,466] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.556141ms
[2022-09-11 05:51:59,498] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr'
[2022-09-11 05:51:59,525] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr%27
[2022-09-11 05:51:59,535] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.455791ms
[2022-09-11 05:51:59,562] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr ON default USING GSI
[2022-09-11 05:51:59,590] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr+ON+default+USING+GSI
[2022-09-11 05:51:59,655] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.309283ms
[2022-09-11 05:51:59,690] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr'
[2022-09-11 05:51:59,718] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0d940eaf47c94deb90c665db42e866fcjob_title_join_yr%27
[2022-09-11 05:51:59,725] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.38937ms
[2022-09-11 05:51:59,725] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'cd5214cc-e957-49cf-9930-76cdd88bfb5f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.38937ms', 'executionTime': '5.331327ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:51:59,834] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:51:59,837] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:51:59,837] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:00,516] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:00,574] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:52:00,574] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:52:00,692] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:52:00,755] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:52:00,758] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:00,758] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:01,422] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:01,478] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:52:01,478] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:52:01,587] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:52:01,642] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:52:01,643] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:52:01,643] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:52:01,669] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:52:01,695] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:52:01,705] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.49007ms
[2022-09-11 05:52:01,734] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:52:01,761] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:52:01,808] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.170187ms
[2022-09-11 05:52:01,891] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:52:01,892] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.8490991159906, 'mem_free': 12848082944, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:52:01,892] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:52:01,896] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:01,897] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:02,558] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:02,563] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:02,563] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:03,214] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:03,220] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:03,220] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:04,142] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:04,149] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:04,150] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:05,232] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:12,310] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-11 05:52:12,450] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:52:12,931] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:52:12,958] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:52:12,959] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:52:13,015] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:52:13,070] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:52:13,122] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:52:13,123] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:52:13,202] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:52:13,203] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:52:13,228] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:52:13,363] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:52:13,364] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:52:13,390] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:52:13,415] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:52:13,415] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:52:13,441] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:52:13,466] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:52:13,467] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:52:13,493] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:52:13,518] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:52:13,518] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:52:13,544] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:52:13,544] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 05:52:13,544] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:52:13,545] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.792s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_9'}
[2022-09-11 05:52:13,642] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:13,642] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:14,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:14,303] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:52:14,381] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:52:14,381] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-11 05:52:14,382] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:52:14,668] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:52:14,696] - [task:165] INFO -  {'uptime': '759', 'memoryTotal': 15466930176, 'memoryFree': 12990226432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:52:14,721] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:52:14,722] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:52:14,722] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:52:14,754] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:52:14,788] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:52:14,788] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:52:14,816] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:52:14,817] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:52:14,817] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:52:14,818] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:52:14,866] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:52:14,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:14,869] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:15,503] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:15,504] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:52:15,630] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:52:15,631] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:52:15,662] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:15,688] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:15,718] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:52:15,846] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:52:15,873] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 12982759424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:52:15,900] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:52:15,930] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:52:15,930] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:52:15,983] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:52:15,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:15,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:16,617] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:16,618] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:52:16,743] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:52:16,745] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:52:16,776] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:16,802] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:16,831] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:52:16,952] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:52:16,980] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 12990758912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:52:17,008] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:52:17,037] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:52:17,037] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:52:17,090] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:52:17,095] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:17,095] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:17,742] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:17,743] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:52:17,869] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:52:17,870] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:52:17,901] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:17,928] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:17,964] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:52:18,084] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:52:18,111] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 12991176704, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:52:18,136] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:52:18,165] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:52:18,165] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:52:18,219] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:52:18,222] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:18,222] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:18,850] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:18,851] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:52:18,971] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:52:18,972] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:52:19,003] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:19,029] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:52:19,057] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:52:19,147] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:52:19,677] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:52:24,682] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:52:24,766] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:52:24,770] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:24,770] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:25,400] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:25,402] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:52:25,527] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:52:25,528] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:52:25,528] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:52:26,314] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:52:26,367] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 05:52:26,368] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:52:27,287] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:52:27,591] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:52:27,873] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:52:27,877] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 05:52:27,936] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:27,936] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:28,632] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:28,637] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:28,637] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:29,640] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:29,648] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:29,648] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:30,745] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:30,754] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:30,754] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:31,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:38,197] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:52:38,197] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.87784697916535, 'mem_free': 13001433088, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:52:38,197] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:52:38,198] - [basetestcase:467] INFO - Time to execute basesetup : 24.55811095237732
[2022-09-11 05:52:38,248] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:52:38,249] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:52:38,302] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:52:38,302] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:52:38,355] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:52:38,355] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:52:38,413] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:52:38,413] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:52:38,463] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:52:38,520] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:52:38,520] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:52:38,520] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:52:43,532] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:52:43,535] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:52:43,535] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:52:44,158] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:52:45,300] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:52:45,472] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:52:47,610] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:52:47,688] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:52:47,689] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:52:47,689] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:53:17,717] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:53:17,746] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:53:17,771] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:53:17,839] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.983338ms
[2022-09-11 05:53:17,839] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b065b5c2-773e-4ad8-a015-db0a8b2b9550', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.983338ms', 'executionTime': '65.914019ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:53:17,839] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:53:17,864] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:53:17,889] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:53:18,612] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 721.569961ms
[2022-09-11 05:53:18,613] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:53:18,673] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:53:18,711] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:53:18,719] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.505416ms
[2022-09-11 05:53:18,925] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:53:18,962] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:53:18,981] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:53:18,982] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:53:18,996] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:53:19,070] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:53:19,891] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee51d15af9c43f4966bdbfceefe6820d34join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:53:19,916] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee51d15af9c43f4966bdbfceefe6820d34join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:53:19,969] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.968425ms
[2022-09-11 05:53:19,970] - [base_gsi:282] INFO - BUILD INDEX on default(`employee51d15af9c43f4966bdbfceefe6820d34join_yr`) USING GSI
[2022-09-11 05:53:21,001] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee51d15af9c43f4966bdbfceefe6820d34join_yr`) USING GSI
[2022-09-11 05:53:21,026] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee51d15af9c43f4966bdbfceefe6820d34join_yr%60%29+USING+GSI
[2022-09-11 05:53:21,057] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.183352ms
[2022-09-11 05:53:21,104] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee51d15af9c43f4966bdbfceefe6820d34join_yr'
[2022-09-11 05:53:21,142] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee51d15af9c43f4966bdbfceefe6820d34join_yr%27
[2022-09-11 05:53:21,152] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.237683ms
[2022-09-11 05:53:22,182] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee51d15af9c43f4966bdbfceefe6820d34join_yr'
[2022-09-11 05:53:22,209] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee51d15af9c43f4966bdbfceefe6820d34join_yr%27
[2022-09-11 05:53:22,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.781977ms
[2022-09-11 05:53:23,253] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee51d15af9c43f4966bdbfceefe6820d34join_yr'
[2022-09-11 05:53:23,278] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee51d15af9c43f4966bdbfceefe6820d34join_yr%27
[2022-09-11 05:53:23,285] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.355466ms
[2022-09-11 05:53:24,316] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:53:24,341] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 05:53:24,345] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.732858ms
[2022-09-11 05:53:24,346] - [base_gsi:504] INFO - {'requestID': 'f1aaaefe-0378-49aa-8c17-7aa300d28c39', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee51d15af9c43f4966bdbfceefe6820d34join_yr', 'index_id': 'ec8a00fac83d466e', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.732858ms', 'executionTime': '2.663236ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 05:53:24,346] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:53:24,347] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:53:24,347] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:53:24,347] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:53:24,348] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:53:24,349] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:53:24,349] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:53:24,349] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:53:24,409] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:53:24,446] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:53:24,471] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 05:53:24,614] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 131.131445ms
[2022-09-11 05:53:24,615] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:53:24,615] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:53:26,945] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:53:26,970] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee51d15af9c43f4966bdbfceefe6820d34join_yr'
[2022-09-11 05:53:26,996] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee51d15af9c43f4966bdbfceefe6820d34join_yr%27
[2022-09-11 05:53:27,002] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.057559ms
[2022-09-11 05:53:27,002] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f715b879-ac97-42a8-97a7-5e197554a226', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.057559ms', 'executionTime': '3.992651ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:53:27,105] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:53:27,108] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:27,108] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:27,862] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:27,917] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:53:27,918] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:53:28,024] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:53:28,082] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:53:28,085] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:28,085] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:28,746] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:28,801] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:53:28,801] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:53:28,907] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:53:28,962] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:53:28,962] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:53:28,963] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:53:28,988] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:53:29,013] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:53:29,022] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.779324ms
[2022-09-11 05:53:29,022] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '23aa862a-fa7a-4d42-ae1a-50711cbff58b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.779324ms', 'executionTime': '6.704344ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:53:29,076] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:53:29,077] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.374242685303788, 'mem_free': 12863639552, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:53:29,077] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:53:29,080] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:29,080] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:29,701] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:29,705] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:29,706] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:30,356] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:30,360] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:30,361] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:31,542] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:31,555] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:31,556] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:32,677] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:39,733] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-11 05:53:39,867] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:53:39,926] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:53:39,978] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:53:40,031] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:53:40,031] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:53:40,106] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:53:40,107] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:53:40,132] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:53:40,260] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:53:40,260] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:53:40,286] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:53:40,311] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:53:40,311] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:53:40,338] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:53:40,363] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:53:40,363] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:53:40,389] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:53:40,415] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:53:40,415] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:53:40,442] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:53:40,442] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 05:53:40,442] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:53:40,443] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.855s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_10'}
[2022-09-11 05:53:40,522] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:40,522] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:41,171] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:41,205] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:53:41,287] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:53:41,287] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-11 05:53:41,287] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:53:41,548] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:53:41,575] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 12846895104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:53:41,600] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:53:41,600] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:53:41,601] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:53:41,637] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:53:41,670] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:53:41,670] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:53:41,698] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:53:41,699] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:53:41,699] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:53:41,699] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:53:41,748] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:53:41,751] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:41,751] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:42,421] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:42,422] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:53:42,551] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:53:42,552] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:53:42,582] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:42,609] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:42,639] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:53:42,769] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:53:42,799] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 12991717376, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:53:42,827] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:53:42,856] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:53:42,856] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:53:42,909] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:53:42,914] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:42,914] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:43,608] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:43,609] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:53:43,747] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:53:43,748] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:53:43,779] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:43,806] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:43,835] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:53:43,958] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:53:43,987] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 12991721472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:53:44,013] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:53:44,041] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:53:44,042] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:53:44,092] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:53:44,095] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:44,095] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:44,741] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:44,742] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:53:44,863] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:53:44,864] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:53:44,894] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:44,925] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:44,952] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:53:45,073] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:53:45,101] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 12991168512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:53:45,126] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:53:45,154] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:53:45,155] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:53:45,205] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:53:45,208] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:45,209] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:45,841] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:45,842] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:53:45,961] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:53:45,962] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:53:45,991] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:46,017] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:53:46,046] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:53:46,131] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:53:46,634] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:53:51,639] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:53:51,724] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:53:51,727] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:51,728] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:52,385] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:52,386] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:53:52,505] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:53:52,506] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:53:52,506] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:53:53,296] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:53:53,350] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 05:53:53,350] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:53:54,262] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:53:54,571] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:53:54,839] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:53:54,842] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 05:53:54,898] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:54,898] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:55,582] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:55,587] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:55,587] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:56,596] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:56,604] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:56,604] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:57,803] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:53:57,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:53:57,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:53:58,990] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:54:05,909] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:54:05,910] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.81886909655138, 'mem_free': 12986449920, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:54:05,910] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:54:05,910] - [basetestcase:467] INFO - Time to execute basesetup : 25.390678644180298
[2022-09-11 05:54:05,960] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:54:05,960] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:54:06,012] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:54:06,012] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:54:06,064] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:54:06,064] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:54:06,116] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:54:06,117] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:54:06,167] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:54:06,227] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:54:06,228] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:54:06,228] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:54:11,236] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:54:11,239] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:54:11,239] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:54:11,888] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:54:12,919] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:54:13,213] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:54:15,325] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:54:15,401] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:54:15,402] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:54:15,402] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:54:45,420] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:54:45,449] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:54:45,474] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:54:45,542] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.214632ms
[2022-09-11 05:54:45,542] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bb4a8325-b984-49c7-a3e2-a5d30774c28d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.214632ms', 'executionTime': '66.129538ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:54:45,542] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:54:45,568] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:54:45,593] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:54:46,248] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 653.550269ms
[2022-09-11 05:54:46,249] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:54:46,329] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:54:46,372] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:54:46,381] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.541233ms
[2022-09-11 05:54:46,579] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:54:46,614] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:54:46,632] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:54:46,633] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:54:46,656] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:54:46,725] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:54:47,544] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee44596015dfff4a708408663440d6a793join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:54:47,571] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee44596015dfff4a708408663440d6a793join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:54:47,647] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 74.562341ms
[2022-09-11 05:54:47,648] - [base_gsi:282] INFO - BUILD INDEX on default(`employee44596015dfff4a708408663440d6a793join_yr`) USING GSI
[2022-09-11 05:54:48,679] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee44596015dfff4a708408663440d6a793join_yr`) USING GSI
[2022-09-11 05:54:48,704] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee44596015dfff4a708408663440d6a793join_yr%60%29+USING+GSI
[2022-09-11 05:54:48,727] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.425672ms
[2022-09-11 05:54:48,785] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee44596015dfff4a708408663440d6a793join_yr'
[2022-09-11 05:54:48,813] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee44596015dfff4a708408663440d6a793join_yr%27
[2022-09-11 05:54:48,822] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.138342ms
[2022-09-11 05:54:49,853] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee44596015dfff4a708408663440d6a793join_yr'
[2022-09-11 05:54:49,881] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee44596015dfff4a708408663440d6a793join_yr%27
[2022-09-11 05:54:49,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.810588ms
[2022-09-11 05:54:50,920] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee44596015dfff4a708408663440d6a793join_yr'
[2022-09-11 05:54:50,947] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee44596015dfff4a708408663440d6a793join_yr%27
[2022-09-11 05:54:50,956] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.581ms
[2022-09-11 05:54:51,988] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:54:52,014] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 05:54:52,018] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.713239ms
[2022-09-11 05:54:52,019] - [base_gsi:504] INFO - {'requestID': 'e9f3c2b6-e37f-4b7d-acce-9067d0f92d96', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee44596015dfff4a708408663440d6a793join_yr', 'index_id': '4353b32725374635', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.713239ms', 'executionTime': '2.642791ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 05:54:52,019] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:54:52,020] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:54:52,020] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:54:52,020] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:54:52,020] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:54:52,021] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:54:52,021] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:54:52,021] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:54:52,082] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:54:52,120] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:54:52,146] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 05:54:52,312] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 152.600662ms
[2022-09-11 05:54:52,312] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:54:52,313] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:54:53,550] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-11 05:54:53,605] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-11 05:54:53,637] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:54:53,665] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 05:54:53,670] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.985498ms
[2022-09-11 05:54:53,670] - [base_gsi:504] INFO - {'requestID': 'efa9be56-780e-4d6b-9bc9-41ff2d1acb61', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee44596015dfff4a708408663440d6a793join_yr', 'index_id': '4353b32725374635', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.985498ms', 'executionTime': '2.911144ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 05:54:53,670] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:54:53,671] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:54:53,671] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:54:53,671] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:54:53,671] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:54:53,671] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:54:53,672] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:54:53,672] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:54:53,725] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:54:53,765] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:54:53,791] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 05:54:53,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 89.955689ms
[2022-09-11 05:54:53,891] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:54:53,891] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:54:55,082] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee44596015dfff4a708408663440d6a793join_yr ON default USING GSI
[2022-09-11 05:54:55,109] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee44596015dfff4a708408663440d6a793join_yr+ON+default+USING+GSI
[2022-09-11 05:54:55,152] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.447151ms
[2022-09-11 05:54:55,188] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee44596015dfff4a708408663440d6a793join_yr'
[2022-09-11 05:54:55,218] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee44596015dfff4a708408663440d6a793join_yr%27
[2022-09-11 05:54:55,226] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.123225ms
[2022-09-11 05:54:55,227] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '50a13280-953b-4444-9156-9894f2954d51', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.123225ms', 'executionTime': '6.060192ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:54:55,337] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:54:55,340] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:54:55,340] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:54:56,064] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:54:56,122] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:54:56,122] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:54:56,240] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:54:56,298] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:54:56,301] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:54:56,301] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:54:56,967] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:54:57,023] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:54:57,024] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:54:57,148] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:54:57,205] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:54:57,206] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:54:57,206] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:54:57,232] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:54:57,261] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:54:57,270] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.825279ms
[2022-09-11 05:54:57,302] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:54:57,331] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:54:57,389] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.2421ms
[2022-09-11 05:54:57,477] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:54:57,478] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.6649399625581, 'mem_free': 12818522112, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:54:57,478] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:54:57,481] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:54:57,482] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:54:58,205] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:54:58,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:54:58,210] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:54:58,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:54:58,927] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:54:58,927] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:00,137] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:00,145] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:00,146] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:01,396] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:09,402] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-11 05:55:09,543] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:55:10,797] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:55:10,826] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:55:10,827] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:55:10,891] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:55:10,944] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:55:10,999] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:55:11,000] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:55:11,082] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:55:11,083] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:55:11,110] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:55:11,243] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:55:11,243] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:55:11,270] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:55:11,295] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:55:11,295] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:55:11,322] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:55:11,348] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:55:11,348] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:55:11,375] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:55:11,400] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:55:11,401] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:55:11,427] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:55:11,427] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 05:55:11,427] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:55:11,428] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 90.959s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_11'}
[2022-09-11 05:55:11,507] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:11,507] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:12,166] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:12,198] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:55:12,276] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:55:12,277] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-11 05:55:12,277] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:55:12,533] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:55:12,560] - [task:165] INFO -  {'uptime': '937', 'memoryTotal': 15466930176, 'memoryFree': 12971859968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:55:12,585] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:55:12,585] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:55:12,585] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:55:12,618] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:55:12,652] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:55:12,652] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:55:12,680] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:55:12,681] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:55:12,681] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:55:12,681] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:55:12,732] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:55:12,735] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:12,735] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:13,449] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:13,450] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:55:13,579] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:55:13,580] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:55:13,611] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:13,638] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:13,667] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:55:13,799] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:55:13,827] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12968009728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:55:13,852] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:55:13,880] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:55:13,880] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:55:13,931] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:55:13,934] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:13,934] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:14,635] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:14,636] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:55:14,774] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:55:14,775] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:55:14,805] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:14,831] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:14,860] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:55:14,980] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:55:15,008] - [task:165] INFO -  {'uptime': '933', 'memoryTotal': 15466930176, 'memoryFree': 12973273088, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:55:15,034] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:55:15,061] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:55:15,062] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:55:15,113] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:55:15,117] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:15,118] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:15,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:15,839] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:55:15,972] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:55:15,973] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:55:16,005] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:16,031] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:16,060] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:55:16,179] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:55:16,208] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12968497152, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:55:16,234] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:55:16,262] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:55:16,262] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:55:16,314] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:55:16,318] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:16,318] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:16,999] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:17,001] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:55:17,124] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:55:17,126] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:55:17,157] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:17,183] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:55:17,213] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:55:17,304] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:55:17,961] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:55:22,962] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:55:23,050] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:55:23,054] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:23,054] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:23,730] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:23,731] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:55:23,855] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:55:23,856] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:55:23,856] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:55:24,442] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:55:24,497] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 05:55:24,497] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:55:25,416] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:55:25,718] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:55:25,972] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:55:25,975] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 05:55:26,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:26,031] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:26,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:26,729] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:26,729] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:27,798] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:27,810] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:27,811] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:29,035] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:29,044] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:29,045] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:30,235] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:37,359] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:55:37,359] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.05663628478212, 'mem_free': 12967096320, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:55:37,360] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:55:37,360] - [basetestcase:467] INFO - Time to execute basesetup : 25.855289459228516
[2022-09-11 05:55:37,411] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:55:37,411] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:55:37,464] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:55:37,465] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:55:37,517] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:55:37,518] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:55:37,569] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:55:37,570] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:55:37,620] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:55:37,685] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:55:37,687] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:55:37,687] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:55:42,697] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:55:42,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:55:42,700] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:55:43,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:55:44,535] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:55:44,709] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:55:47,824] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:55:47,902] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:55:47,902] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:55:47,902] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:56:17,912] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:56:17,941] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:56:17,966] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:56:18,034] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.617106ms
[2022-09-11 05:56:18,034] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '501a3961-722f-4639-b57a-83f947fabd4b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.617106ms', 'executionTime': '66.556931ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:56:18,035] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:56:18,061] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:56:18,086] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:56:18,810] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 721.606123ms
[2022-09-11 05:56:18,810] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:56:18,866] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:56:18,904] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:56:18,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.356827ms
[2022-09-11 05:56:19,108] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:56:19,143] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:56:19,164] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:56:19,173] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:56:19,189] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:56:19,255] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:56:20,068] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec709aea7633f40acaf6e03a60cc1075cjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 05:56:20,093] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec709aea7633f40acaf6e03a60cc1075cjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 05:56:20,170] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 74.575538ms
[2022-09-11 05:56:20,170] - [base_gsi:282] INFO - BUILD INDEX on default(`employeec709aea7633f40acaf6e03a60cc1075cjoin_yr`) USING GSI
[2022-09-11 05:56:21,201] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeec709aea7633f40acaf6e03a60cc1075cjoin_yr`) USING GSI
[2022-09-11 05:56:21,227] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeec709aea7633f40acaf6e03a60cc1075cjoin_yr%60%29+USING+GSI
[2022-09-11 05:56:21,249] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.492919ms
[2022-09-11 05:56:21,299] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec709aea7633f40acaf6e03a60cc1075cjoin_yr'
[2022-09-11 05:56:21,328] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec709aea7633f40acaf6e03a60cc1075cjoin_yr%27
[2022-09-11 05:56:21,337] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.280551ms
[2022-09-11 05:56:22,366] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec709aea7633f40acaf6e03a60cc1075cjoin_yr'
[2022-09-11 05:56:22,393] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec709aea7633f40acaf6e03a60cc1075cjoin_yr%27
[2022-09-11 05:56:22,404] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.68554ms
[2022-09-11 05:56:23,435] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec709aea7633f40acaf6e03a60cc1075cjoin_yr'
[2022-09-11 05:56:23,460] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec709aea7633f40acaf6e03a60cc1075cjoin_yr%27
[2022-09-11 05:56:23,469] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.852573ms
[2022-09-11 05:56:24,500] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:56:24,526] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 05:56:24,531] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.683783ms
[2022-09-11 05:56:24,531] - [base_gsi:504] INFO - {'requestID': '54a67e9b-ed12-4616-97c7-ec2406ae8a4c', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec709aea7633f40acaf6e03a60cc1075cjoin_yr', 'index_id': '7347d9e2d77ba54c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.683783ms', 'executionTime': '2.6113ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 05:56:24,531] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 05:56:24,532] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:56:24,532] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 05:56:24,532] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 05:56:24,532] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:56:24,533] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:56:24,533] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:56:24,533] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 05:56:24,591] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 05:56:24,629] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 05:56:24,654] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 05:56:24,828] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 160.801287ms
[2022-09-11 05:56:24,829] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 05:56:24,829] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 05:56:27,006] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 05:56:29,557] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:56:29,618] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 05:56:29,618] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:56:30,527] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:56:30,867] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:56:31,156] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:56:31,159] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 05:56:33,193] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:56:33,193] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:33,221] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:56:33,249] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:56:33,250] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:33,277] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:56:33,306] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:56:33,307] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:33,336] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:56:33,364] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:56:33,364] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:33,392] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:56:33,445] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:56:33,472] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec709aea7633f40acaf6e03a60cc1075cjoin_yr'
[2022-09-11 05:56:33,498] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec709aea7633f40acaf6e03a60cc1075cjoin_yr%27
[2022-09-11 05:56:33,545] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.79553ms
[2022-09-11 05:56:33,545] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b40cac99-6082-4927-b425-9a5b17ce881f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '44.79553ms', 'executionTime': '44.730861ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:56:33,572] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec709aea7633f40acaf6e03a60cc1075cjoin_yr'
[2022-09-11 05:56:33,598] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec709aea7633f40acaf6e03a60cc1075cjoin_yr%27
[2022-09-11 05:56:33,601] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.38399ms
[2022-09-11 05:56:33,601] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'dc2badde-6318-4c8e-8247-93122f461e0b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.38399ms', 'executionTime': '1.31999ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:56:33,654] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:56:33,783] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:56:33,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:33,785] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:34,515] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:34,572] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:56:34,573] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:56:34,692] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:56:34,749] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:56:34,752] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:34,752] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:35,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:35,512] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:56:35,512] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:56:35,636] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:56:35,695] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:56:35,695] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:56:35,695] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:56:35,720] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:56:35,746] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:56:35,755] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.310902ms
[2022-09-11 05:56:35,756] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '98c4ce37-cd34-4618-808c-8d25c4e08e69', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.310902ms', 'executionTime': '8.249628ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:56:35,756] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:56:35,782] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:56:35,808] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:56:35,811] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.365153ms
[2022-09-11 05:56:35,811] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '57d9c3e6-8e8e-46ab-b524-525f6d8abcf2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.365153ms', 'executionTime': '1.308301ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:56:35,863] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:56:35,864] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.48957806680149, 'mem_free': 12803039232, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:56:35,864] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:56:35,867] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:35,867] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:36,556] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:36,561] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:36,561] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:37,375] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:37,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:37,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:38,600] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:38,608] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:38,608] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:39,816] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:47,491] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 05:56:47,635] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:56:47,942] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:56:47,969] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:56:47,969] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:56:48,023] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:56:48,076] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:56:48,128] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:56:48,129] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:56:48,208] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:56:48,209] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:56:48,234] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:48,362] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:56:48,362] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:48,388] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:56:48,414] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:56:48,415] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:48,441] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:56:48,466] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:56:48,466] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:48,492] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:56:48,518] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:56:48,518] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:48,544] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:56:48,544] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 05:56:48,544] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:56:48,545] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 97.092s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_05-39-31/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-11 05:56:48,603] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:48,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:49,301] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:49,335] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:56:49,413] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 05:56:49,414] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-11 05:56:49,414] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 05:56:49,629] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 05:56:49,655] - [task:165] INFO -  {'uptime': '1034', 'memoryTotal': 15466930176, 'memoryFree': 12958867456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:56:49,681] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 05:56:49,681] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 05:56:49,681] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 05:56:49,715] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 05:56:49,749] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 05:56:49,750] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 05:56:49,777] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 05:56:49,778] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 05:56:49,779] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 05:56:49,779] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 05:56:49,826] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:56:49,829] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:49,829] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:50,512] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:50,513] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:56:50,642] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:56:50,643] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:56:50,676] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:50,706] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:50,736] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:56:50,862] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 05:56:50,888] - [task:165] INFO -  {'uptime': '1035', 'memoryTotal': 15466930176, 'memoryFree': 12958937088, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:56:50,914] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:56:50,942] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 05:56:50,942] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 05:56:50,992] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:56:50,995] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:50,995] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:51,713] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:51,714] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:56:51,853] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:56:51,854] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:56:51,887] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:51,915] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:51,947] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:56:52,062] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 05:56:52,088] - [task:165] INFO -  {'uptime': '1034', 'memoryTotal': 15466930176, 'memoryFree': 12958556160, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:56:52,114] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:56:52,140] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 05:56:52,141] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 05:56:52,191] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:56:52,194] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:52,195] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:52,900] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:52,901] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:56:53,043] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:56:53,044] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:56:53,074] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:53,101] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:53,130] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:56:53,252] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 05:56:53,281] - [task:165] INFO -  {'uptime': '1030', 'memoryTotal': 15466930176, 'memoryFree': 12958781440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 05:56:53,308] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 05:56:53,337] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 05:56:53,337] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 05:56:53,390] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 05:56:53,393] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:56:53,393] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:56:54,101] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:56:54,102] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:56:54,230] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:56:54,231] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:56:54,260] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:54,285] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 05:56:54,313] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-11 05:56:54,403] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 05:56:54,926] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:56:59,931] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 05:57:00,019] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 05:57:00,023] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:00,023] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:00,761] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:00,762] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 05:57:00,899] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 05:57:00,900] - [remote_util:5249] INFO - b'ok'
[2022-09-11 05:57:00,901] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 05:57:01,540] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 05:57:01,596] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 05:57:01,596] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 05:57:02,501] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:57:02,827] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:57:03,087] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 05:57:03,091] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-11 05:57:03,147] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:03,147] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:03,885] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:03,890] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:03,890] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:05,057] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:05,066] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:05,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:06,306] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:06,315] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:06,315] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:07,568] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:14,689] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:57:14,689] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.31940254956212, 'mem_free': 12956741632, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:57:14,690] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:57:14,690] - [basetestcase:467] INFO - Time to execute basesetup : 26.09003758430481
[2022-09-11 05:57:14,740] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:57:14,740] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:57:14,793] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:57:14,793] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:57:14,846] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:57:14,846] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:57:14,897] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 05:57:14,898] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 05:57:14,948] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:57:15,008] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-11 05:57:15,008] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 05:57:15,008] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 05:57:20,016] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 05:57:20,018] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:20,019] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:20,719] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:21,901] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 05:57:22,080] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 05:57:24,219] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 05:57:24,300] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:57:24,300] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 05:57:24,300] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 05:57:54,324] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 05:57:54,354] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:57:54,380] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:57:54,450] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.726793ms
[2022-09-11 05:57:54,450] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '15d4dd9c-799f-4317-a2fc-62e75d138240', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.726793ms', 'executionTime': '67.660593ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 05:57:54,450] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 05:57:54,477] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 05:57:54,502] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 05:57:55,211] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 706.917441ms
[2022-09-11 05:57:55,211] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 05:57:55,287] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:57:55,317] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:57:55,326] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.42281ms
[2022-09-11 05:57:55,528] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:57:55,568] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 05:57:55,585] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 05:57:55,586] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 05:57:55,605] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 05:57:55,666] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:57:55,807] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:57:55,810] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:55,811] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:56,582] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:56,639] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:57:56,640] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 05:57:56,760] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:57:56,820] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:57:56,823] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:56,823] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:57,625] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:57,680] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 05:57:57,681] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 05:57:57,814] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 05:57:57,873] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 05:57:57,873] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 05:57:57,873] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:57:57,900] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 05:57:57,926] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 05:57:57,933] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.38759ms
[2022-09-11 05:57:57,960] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 05:57:57,987] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 05:57:58,032] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.72532ms
[2022-09-11 05:57:58,111] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 05:57:58,111] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.90653064064958, 'mem_free': 12753694720, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 05:57:58,111] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 05:57:58,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:58,116] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:57:58,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:57:58,915] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:57:58,915] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:58:00,244] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:58:00,252] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:58:00,252] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:58:01,629] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:58:01,638] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 05:58:01,638] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 05:58:03,040] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 05:58:10,833] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-11 05:58:10,979] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 05:58:12,051] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 05:58:12,080] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 05:58:12,080] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 05:58:12,153] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:58:12,208] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:58:12,260] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 05:58:12,261] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 05:58:12,346] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 05:58:12,347] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 05:58:12,373] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:58:12,507] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 05:58:12,508] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:58:12,535] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 05:58:12,560] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 05:58:12,560] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:58:12,587] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 05:58:12,613] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 05:58:12,613] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:58:12,640] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 05:58:12,666] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 05:58:12,667] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 05:58:12,694] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 05:58:12,694] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-11 05:58:12,694] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 05:58:12,695] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 84.178s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 31688 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-11.09.2022-04.09.cfg
Build Log: make-11.09.2022-04.09.log
Server Log: logs-11.09.2022-04.09.tar.gz

Finished