Building

Started building at 2022/09/11 00:06:33
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 11.09.2022-00.06
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: 
 4da8ba0f MB-51825: Remove vbuckets arg from indexer command line and its usage
Switching plasma to unstable
plasma is at 0760e59, changes since last good build: 
fatal: Invalid revision range 67d5c96bc6a9c01fadbcc5dbdd2a62a39593b413..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable ec4c2433] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 152090f4] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable d12ed479] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable 9bf886d7] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable a95d2ee1] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 90b50d37] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable bc9b64ec] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable 0b31bcd1] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable a97da72e] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 325b5a72] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable 15809ad5] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 2a65c018] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable fda2a968] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 0e2f2c3] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable 2595765] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 0b65f1b] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 6763f76] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 393be9a] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable 835ca12] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable e859b79] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 5da4de5] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/11 01:00:32
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Sep 11 01:05:55 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 01:09:47 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 01:10:27 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 11 01:11:30 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 11 01:12:17 b" 'stop_on_failure': 'True'}"
Sep 11 01:12:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 01:12:17 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_1'}"
Sep 11 01:12:17 b'-->result: '
Sep 11 01:12:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 11 01:12:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 01:12:17 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_2'}"
Sep 11 01:12:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 01:12:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 11 01:12:17 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_3'}"
Sep 11 01:12:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 01:12:17 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 01:12:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 11 01:12:17 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_4'}"
Sep 11 01:12:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 01:12:17 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 01:12:17 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 01:22:42 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 11 01:23:27 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 11 01:27:12 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 01:31:09 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 01:33:52 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 01:33:52 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_5'}"
Sep 11 01:33:52 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 01:33:52 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 11 01:33:52 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 11 01:33:52 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_6'}"
Sep 11 01:33:52 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 01:33:52 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 01:33:52 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 11 01:33:52 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_7'}"
Sep 11 01:33:52 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 01:33:52 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 01:33:52 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 11 01:33:52 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 11 01:33:52 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-00-53/test_8'}"
Sep 11 01:33:52 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 01:33:52 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 01:33:52 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 01:33:52 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 11 01:33:52 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 11 01:33:53 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 01:33:53 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 01:33:53 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 11 01:33:53 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 11 01:33:53 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 11 01:33:53 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 11 01:33:53 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 11 01:33:53 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 180: go: command not found
/home/buildbot/bin/dotest: line 183: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-11 01:36:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:36:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:36:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:36:45 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 01:36:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 01:36:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 01:37:14 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:14 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-11 01:37:15 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 2065223680, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:37:15 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:37:15 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:37:16 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1880514560, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:37:16 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:37:17 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1770000384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:37:17 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:37:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:37:18 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '19', 'memoryTotal': 15466930176, 'memoryFree': 1673338880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:37:18 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:37:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:37:18 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 01:37:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 01:37:18 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-11 01:37:19 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:37:24 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 01:37:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 01:37:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-11 01:37:24 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 01:37:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 01:37:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 01:37:26 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 0, 'mem_free': 13391765504, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.72797703742981
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:37:29 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 01:37:30 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 01:37:35 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 01:37:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:37:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:37:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:37:36 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 01:37:36 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 01:37:38 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 01:37:38 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:37:38 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 01:37:38 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 01:38:08 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 01:38:08 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 01:38:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 01:38:08 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 134.016106ms
2022-09-11 01:38:08 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '8a27af9b-99f8-43d5-ad55-9a180e82d031', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '134.016106ms', 'executionTime': '133.799184ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 01:38:08 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 01:38:08 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 01:38:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 667.259292ms
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.007755ms
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.206063ms
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 42.656504ms
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.770737730145084, 'mem_free': 13358067712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:15 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 01:38:15 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 01:38:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 01:38:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 01:38:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 92.040s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 01:38:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.965606079029664, 'mem_free': 13351936000, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 01:38:22 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 01:38:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 01:38:23 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13321494528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 01:38:24 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13334413312, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 01:38:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13334888448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:38:25 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13332332544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 01:38:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 01:38:26 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 01:38:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 01:38:32 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 01:38:33 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 01:38:33 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-11 01:38:33 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 01:38:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 01:38:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 01:38:35 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.81496191128983, 'mem_free': 13312880640, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 22.273450136184692
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 01:38:39 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 01:38:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 01:38:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:38:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:38:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:38:45 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 01:38:46 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 01:38:47 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 01:38:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:38:47 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 01:38:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 01:39:17 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 01:39:17 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 67.042037ms
2022-09-11 01:39:18 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '3139e30f-fba0-4b2a-b863-503766f490aa', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.042037ms', 'executionTime': '66.961082ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 656.144383ms
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 01:39:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.610608ms
2022-09-11 01:39:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:39:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 01:39:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 01:39:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 01:39:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 01:39:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:39:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee47aea008987b4127b67224e1818a8b4djob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-11 01:39:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee47aea008987b4127b67224e1818a8b4djob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-11 01:39:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 28.555512ms
2022-09-11 01:39:20 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee47aea008987b4127b67224e1818a8b4djob_title) USING GSI
2022-09-11 01:39:21 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee47aea008987b4127b67224e1818a8b4djob_title) USING GSI
2022-09-11 01:39:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee47aea008987b4127b67224e1818a8b4djob_title%29+USING+GSI
2022-09-11 01:39:21 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 25.227419ms
2022-09-11 01:39:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee47aea008987b4127b67224e1818a8b4djob_title'
2022-09-11 01:39:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee47aea008987b4127b67224e1818a8b4djob_title%27
2022-09-11 01:39:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.418252ms
2022-09-11 01:39:23 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee47aea008987b4127b67224e1818a8b4djob_title'
2022-09-11 01:39:23 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee47aea008987b4127b67224e1818a8b4djob_title%27
2022-09-11 01:39:23 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.679297ms
2022-09-11 01:39:24 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 01:39:24 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 01:39:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-11 01:39:24 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.252548ms
2022-09-11 01:39:24 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '01db428d-0c24-4202-85d0-e3907a697a0a', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee47aea008987b4127b67224e1818a8b4djob_title', 'index_id': 'd92de52123bb37c8', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.252548ms', 'executionTime': '2.179701ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-11 01:39:24 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 01:39:24 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 01:39:24 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 01:39:24 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-11 01:39:24 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-11 01:39:24 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-11 01:39:24 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-11 01:39:24 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-11 01:39:24 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-11 01:39:25 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 01:39:25 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 01:39:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-11 01:39:25 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 127.093454ms
2022-09-11 01:39:25 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-11 01:39:25 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-11 01:39:26 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 01:39:26 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee47aea008987b4127b67224e1818a8b4djob_title'
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee47aea008987b4127b67224e1818a8b4djob_title%27
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.715036ms
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee47aea008987b4127b67224e1818a8b4djob_title ON default USING GSI
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee47aea008987b4127b67224e1818a8b4djob_title+ON+default+USING+GSI
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 40.384542ms
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee47aea008987b4127b67224e1818a8b4djob_title'
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee47aea008987b4127b67224e1818a8b4djob_title%27
2022-09-11 01:39:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.236919ms
2022-09-11 01:39:27 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '42e2a05e-8b00-4960-9cba-921170f30172', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.236919ms', 'executionTime': '5.169624ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 01:39:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:39:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:39:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.510273ms
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 53.213295ms
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.415417841770966, 'mem_free': 13250330624, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:39:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 01:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 01:39:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 01:39:33 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 01:39:33 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 01:39:34 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 01:39:34 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.672s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_2'}
[2022-09-11 01:39:34,882] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:34,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:35,135] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:35,167] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:39:35,246] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:39:35,247] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-11 01:39:35,247] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:39:35,911] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:39:35,938] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13215424512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:39:35,964] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:39:35,964] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:39:35,965] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:39:35,999] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:39:36,037] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:39:36,037] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:39:36,066] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:39:36,067] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:39:36,067] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:39:36,067] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:39:36,122] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:39:36,125] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:36,125] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:36,387] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:36,388] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:39:36,457] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:39:36,457] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:39:36,487] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:39:36,487] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:39:36,615] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:39:36,642] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13245952000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:39:36,668] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:39:36,695] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:39:36,695] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:39:36,746] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:39:36,749] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:36,749] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:37,014] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:37,015] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:39:37,087] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:39:37,088] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:39:37,118] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:39:37,119] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:39:37,244] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:39:37,272] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13200957440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:39:37,299] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:39:37,326] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:39:37,327] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:39:37,378] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:39:37,382] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:37,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:37,665] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:37,666] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:39:37,733] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:39:37,734] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:39:37,765] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:39:37,766] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:39:37,880] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:39:37,907] - [task:165] INFO -  {'uptime': '160', 'memoryTotal': 15466930176, 'memoryFree': 13219713024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:39:37,933] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:39:37,960] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:39:37,961] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:39:38,010] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:39:38,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:38,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:38,264] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:38,265] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:39:38,332] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:39:38,333] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:39:38,362] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:39:38,363] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:39:38,451] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:39:39,084] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:39:44,089] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:39:44,172] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:39:44,177] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:44,178] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:44,451] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:44,452] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:39:44,519] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:39:44,520] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:39:44,520] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:39:45,587] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:39:45,641] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 01:39:45,641] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:39:46,630] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:39:46,930] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:39:47,251] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:39:47,254] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:39:47,318] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:47,319] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:47,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:47,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:47,578] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:47,823] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:47,828] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:47,828] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:48,067] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:48,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:48,072] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:48,363] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:51,962] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:39:51,962] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.86678046824219, 'mem_free': 13228867584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:39:51,963] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:39:51,963] - [basetestcase:467] INFO - Time to execute basesetup : 17.08292531967163
[2022-09-11 01:39:52,013] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:39:52,014] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:39:52,067] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:39:52,067] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:39:52,120] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:39:52,120] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:39:52,172] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:39:52,172] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:39:52,222] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:39:52,283] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:39:52,283] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:39:52,283] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:39:57,295] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:39:57,298] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:39:57,298] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:39:57,547] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:39:58,482] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:39:58,654] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:40:00,338] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:40:00,362] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 01:40:01,025] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:40:01,025] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:40:01,025] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:40:31,047] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:40:31,075] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:40:31,100] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:40:31,170] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.776423ms
[2022-09-11 01:40:31,170] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c676e8db-2828-43dd-ba29-a32d78a59be3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.776423ms', 'executionTime': '67.70535ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:40:31,171] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:40:31,197] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:40:31,222] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:40:31,941] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 717.504387ms
[2022-09-11 01:40:31,942] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:40:31,999] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:40:32,038] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:40:32,046] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.513741ms
[2022-09-11 01:40:32,234] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:40:32,268] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:40:32,281] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:40:32,282] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:40:32,295] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:40:32,375] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:40:33,199] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5756609006204cdcab794e95f9571b7bjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:40:33,225] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5756609006204cdcab794e95f9571b7bjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:40:33,267] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.757656ms
[2022-09-11 01:40:33,268] - [base_gsi:282] INFO - BUILD INDEX on default(employee5756609006204cdcab794e95f9571b7bjob_title) USING GSI
[2022-09-11 01:40:34,299] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5756609006204cdcab794e95f9571b7bjob_title) USING GSI
[2022-09-11 01:40:34,324] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee5756609006204cdcab794e95f9571b7bjob_title%29+USING+GSI
[2022-09-11 01:40:34,347] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.026155ms
[2022-09-11 01:40:35,378] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5756609006204cdcab794e95f9571b7bjob_title'
[2022-09-11 01:40:35,403] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5756609006204cdcab794e95f9571b7bjob_title%27
[2022-09-11 01:40:35,413] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.548748ms
[2022-09-11 01:40:36,444] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5756609006204cdcab794e95f9571b7bjob_title'
[2022-09-11 01:40:36,471] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5756609006204cdcab794e95f9571b7bjob_title%27
[2022-09-11 01:40:36,478] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.02821ms
[2022-09-11 01:40:36,932] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 01:40:37,106] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:40:38,009] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:40:38,481] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:40:38,515] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:40:38,543] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 01:40:38,547] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.524966ms
[2022-09-11 01:40:38,547] - [task:3246] INFO - {'requestID': '81ec2c15-3ad2-4969-9bd2-7e87b6d69f4c', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5756609006204cdcab794e95f9571b7bjob_title', 'index_id': 'b7d35399c0885b2d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.524966ms', 'executionTime': '2.447721ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 01:40:38,548] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:40:38,548] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:40:38,548] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:40:38,549] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:40:38,549] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 01:40:38,549] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:40:38,549] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:40:38,550] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:40:38,550] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:40:39,549] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:40:39,580] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:40:39,606] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 01:40:39,730] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 114.039425ms
[2022-09-11 01:40:39,730] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:40:39,730] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:40:40,526] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:40:40,526] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:40:41,557] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5756609006204cdcab794e95f9571b7bjob_title'
[2022-09-11 01:40:41,582] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5756609006204cdcab794e95f9571b7bjob_title%27
[2022-09-11 01:40:41,591] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.402407ms
[2022-09-11 01:40:41,617] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5756609006204cdcab794e95f9571b7bjob_title ON default USING GSI
[2022-09-11 01:40:41,642] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5756609006204cdcab794e95f9571b7bjob_title+ON+default+USING+GSI
[2022-09-11 01:40:41,687] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.934409ms
[2022-09-11 01:40:41,720] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5756609006204cdcab794e95f9571b7bjob_title'
[2022-09-11 01:40:41,746] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5756609006204cdcab794e95f9571b7bjob_title%27
[2022-09-11 01:40:41,753] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.634991ms
[2022-09-11 01:40:41,753] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e6bc1946-fbba-41a1-9eed-e31727da5132', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.634991ms', 'executionTime': '5.571297ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:40:41,857] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:40:41,860] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:41,860] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:42,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:42,249] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:40:42,250] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:40:42,309] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:40:42,363] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:40:42,366] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:42,367] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:42,710] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:42,774] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:40:42,775] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:40:42,839] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:40:42,893] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:40:42,894] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:40:42,894] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:40:42,923] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:40:42,948] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:40:42,958] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.946785ms
[2022-09-11 01:40:42,984] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:40:43,012] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:40:43,075] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.375821ms
[2022-09-11 01:40:43,135] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:40:43,136] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.35919273728679, 'mem_free': 13104693248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:40:43,136] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:40:43,140] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:43,140] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:43,478] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:43,483] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:43,483] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:43,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:43,843] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:43,843] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:44,338] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:44,345] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:44,345] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:44,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:48,860] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:40:49,005] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:40:50,156] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:40:50,184] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:40:50,184] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:40:50,243] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:40:50,296] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:40:50,349] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:40:50,350] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:40:50,432] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:40:50,433] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:40:50,460] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:40:50,593] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:40:50,594] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:40:50,624] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:40:50,650] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:40:50,651] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:40:50,680] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:40:50,706] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:40:50,707] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:40:50,736] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:40:50,761] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:40:50,762] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:40:50,788] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:40:50,788] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:40:50,788] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:40:50,788] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 75.964s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_3'}
[2022-09-11 01:40:50,892] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:50,893] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:51,210] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:51,241] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:40:51,320] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:40:51,321] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-11 01:40:51,321] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:40:51,922] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:40:51,950] - [task:165] INFO -  {'uptime': '242', 'memoryTotal': 15466930176, 'memoryFree': 13179961344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:40:51,976] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:40:51,976] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:40:51,976] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:40:52,004] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:40:52,038] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:40:52,039] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:40:52,067] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:40:52,067] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:40:52,068] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:40:52,068] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:40:52,116] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:40:52,119] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:52,119] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:52,412] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:52,414] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:40:52,488] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:40:52,489] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:40:52,519] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:40:52,520] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:40:52,644] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:40:52,670] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13108645888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:40:52,696] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:40:52,723] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:40:52,724] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:40:52,777] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:40:52,780] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:52,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:53,099] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:53,101] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:40:53,173] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:40:53,174] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:40:53,202] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:40:53,203] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:40:53,318] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:40:53,345] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13179617280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:40:53,371] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:40:53,399] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:40:53,399] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:40:53,447] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:40:53,453] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:53,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:53,750] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:53,751] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:40:53,823] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:40:53,825] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:40:53,853] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:40:53,854] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:40:53,975] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:40:54,002] - [task:165] INFO -  {'uptime': '235', 'memoryTotal': 15466930176, 'memoryFree': 13121761280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:40:54,027] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:40:54,054] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:40:54,054] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:40:54,105] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:40:54,108] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:40:54,108] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:40:54,403] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:40:54,404] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:40:54,476] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:40:54,478] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:40:54,508] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:40:54,508] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:40:54,600] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:40:55,140] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:41:00,145] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:41:00,235] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:41:00,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:41:00,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:41:00,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:41:00,575] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:41:00,659] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:41:00,660] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:41:00,660] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:41:01,722] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:41:01,778] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 01:41:01,779] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:41:02,672] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:41:02,992] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:41:03,264] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:41:03,267] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:41:03,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:41:03,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:41:03,642] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:41:03,647] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:41:03,647] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:41:03,972] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:41:03,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:41:03,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:41:04,306] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:41:04,318] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:41:04,318] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:41:04,887] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:41:08,885] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:41:08,886] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.36380133372517, 'mem_free': 13177786368, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:41:08,886] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:41:08,886] - [basetestcase:467] INFO - Time to execute basesetup : 17.99630379676819
[2022-09-11 01:41:08,938] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:41:08,938] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:41:08,991] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:41:08,991] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:41:09,043] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:41:09,044] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:41:09,099] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:41:09,099] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:41:09,150] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:41:09,211] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:41:09,211] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:41:09,211] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:41:14,223] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:41:14,226] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:41:14,226] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:41:14,540] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:41:15,499] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:41:15,667] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:41:18,301] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:41:18,317] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 01:41:19,155] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:41:19,155] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:41:19,155] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:41:49,165] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:41:49,194] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:41:49,220] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:41:49,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 71.704186ms
[2022-09-11 01:41:49,294] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2bf6a64e-f275-4d99-bb3c-b822b572e402', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '71.704186ms', 'executionTime': '71.573095ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:41:49,294] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:41:49,320] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:41:49,345] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:41:49,982] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 635.447017ms
[2022-09-11 01:41:49,983] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:41:50,039] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:41:50,080] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:41:50,089] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.091125ms
[2022-09-11 01:41:50,276] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:41:50,315] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:41:50,333] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:41:50,333] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:41:50,358] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:41:50,440] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:41:51,243] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeefc536e72e40f43a78ad731aff1c83d21job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:41:51,269] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeefc536e72e40f43a78ad731aff1c83d21job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:41:51,310] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.347957ms
[2022-09-11 01:41:51,311] - [base_gsi:282] INFO - BUILD INDEX on default(employeefc536e72e40f43a78ad731aff1c83d21job_title) USING GSI
[2022-09-11 01:41:52,341] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeefc536e72e40f43a78ad731aff1c83d21job_title) USING GSI
[2022-09-11 01:41:52,368] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeefc536e72e40f43a78ad731aff1c83d21job_title%29+USING+GSI
[2022-09-11 01:41:52,392] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.39115ms
[2022-09-11 01:41:53,423] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefc536e72e40f43a78ad731aff1c83d21job_title'
[2022-09-11 01:41:53,449] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefc536e72e40f43a78ad731aff1c83d21job_title%27
[2022-09-11 01:41:53,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.965905ms
[2022-09-11 01:41:54,489] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefc536e72e40f43a78ad731aff1c83d21job_title'
[2022-09-11 01:41:54,514] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefc536e72e40f43a78ad731aff1c83d21job_title%27
[2022-09-11 01:41:54,524] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.909897ms
[2022-09-11 01:41:54,912] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 01:41:55,161] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:41:55,952] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:41:56,527] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:41:56,559] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:41:56,585] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 01:41:56,590] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.908343ms
[2022-09-11 01:41:56,590] - [task:3246] INFO - {'requestID': 'ad1701b7-cf9e-4faa-8791-642f31e347a4', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeefc536e72e40f43a78ad731aff1c83d21job_title', 'index_id': '9f4c269a2974310b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.908343ms', 'executionTime': '2.843274ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 01:41:56,590] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:41:56,591] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:41:56,591] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:41:56,592] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:41:56,592] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 01:41:56,592] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:41:56,592] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:41:56,593] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:41:56,593] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:41:57,591] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:41:57,624] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:41:57,650] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 01:41:57,794] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 133.041653ms
[2022-09-11 01:41:57,794] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:41:57,795] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:41:58,591] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:41:58,591] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:41:59,623] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefc536e72e40f43a78ad731aff1c83d21job_title'
[2022-09-11 01:41:59,649] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefc536e72e40f43a78ad731aff1c83d21job_title%27
[2022-09-11 01:41:59,659] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.095234ms
[2022-09-11 01:41:59,687] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeefc536e72e40f43a78ad731aff1c83d21job_title ON default USING GSI
[2022-09-11 01:41:59,713] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeefc536e72e40f43a78ad731aff1c83d21job_title+ON+default+USING+GSI
[2022-09-11 01:41:59,758] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.315309ms
[2022-09-11 01:41:59,791] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefc536e72e40f43a78ad731aff1c83d21job_title'
[2022-09-11 01:41:59,817] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefc536e72e40f43a78ad731aff1c83d21job_title%27
[2022-09-11 01:41:59,824] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.53123ms
[2022-09-11 01:41:59,824] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ec856eb2-0df3-4510-9faf-a88b9295432e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.53123ms', 'executionTime': '5.467174ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:41:59,928] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:41:59,932] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:41:59,932] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:00,334] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:00,391] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:42:00,392] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:42:00,460] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:42:00,516] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:42:00,519] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:00,519] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:00,937] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:00,998] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:42:00,998] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:42:01,068] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:42:01,124] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:42:01,124] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:42:01,125] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:42:01,150] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:42:01,176] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:42:01,183] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.862691ms
[2022-09-11 01:42:01,210] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:42:01,237] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:42:01,296] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.60498ms
[2022-09-11 01:42:01,370] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:42:01,370] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.14584093752281, 'mem_free': 13044068352, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:42:01,371] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:42:01,375] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:01,375] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:01,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:01,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:01,782] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:02,347] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:02,354] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:02,354] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:03,048] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:03,055] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:03,056] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:03,746] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:07,952] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:42:08,099] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:42:09,260] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:42:09,289] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:42:09,289] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:42:09,355] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:42:09,408] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:42:09,462] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:42:09,463] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:42:09,544] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:42:09,544] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:42:09,571] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:42:09,704] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:42:09,704] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:42:09,732] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:42:09,757] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:42:09,757] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:42:09,783] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:42:09,808] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:42:09,808] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:42:09,835] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:42:09,860] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:42:09,861] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:42:09,888] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:42:09,888] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:42:09,888] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:42:09,889] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.050s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_4'}
[2022-09-11 01:42:09,988] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:09,988] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:10,352] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:10,382] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:42:10,461] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:42:10,461] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-11 01:42:10,462] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:42:11,016] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:42:11,043] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13134598144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:42:11,068] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:42:11,068] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:42:11,069] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:42:11,097] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:42:11,140] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:42:11,141] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:42:11,170] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:42:11,171] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:42:11,171] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:42:11,171] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:42:11,219] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:42:11,222] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:11,222] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:11,580] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:11,581] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:42:11,663] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:42:11,664] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:42:11,694] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:42:11,695] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:42:11,815] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:42:11,841] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13136687104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:42:11,868] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:42:11,895] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:42:11,895] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:42:11,944] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:42:11,948] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:11,948] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:12,304] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:12,305] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:42:12,386] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:42:12,387] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:42:12,416] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:42:12,417] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:42:12,533] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:42:12,559] - [task:165] INFO -  {'uptime': '317', 'memoryTotal': 15466930176, 'memoryFree': 13137465344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:42:12,585] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:42:12,615] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:42:12,616] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:42:12,666] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:42:12,670] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:12,671] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:13,038] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:13,039] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:42:13,121] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:42:13,122] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:42:13,151] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:42:13,152] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:42:13,273] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:42:13,299] - [task:165] INFO -  {'uptime': '316', 'memoryTotal': 15466930176, 'memoryFree': 13136031744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:42:13,325] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:42:13,353] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:42:13,354] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:42:13,403] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:42:13,406] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:13,406] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:13,770] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:13,771] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:42:13,854] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:42:13,855] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:42:13,889] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:42:13,890] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:42:13,986] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:42:14,561] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:42:19,566] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:42:19,651] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:42:19,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:19,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:20,047] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:20,048] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:42:20,129] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:42:20,130] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:42:20,131] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:42:21,099] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:42:21,156] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 01:42:21,157] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:42:22,074] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:42:22,382] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:42:22,661] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:42:22,663] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:42:22,717] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:22,717] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:23,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:23,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:23,082] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:23,435] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:23,440] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:23,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:23,908] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:23,922] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:23,922] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:24,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:29,228] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:42:29,229] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.72698275706888, 'mem_free': 13138329600, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:42:29,229] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:42:29,229] - [basetestcase:467] INFO - Time to execute basesetup : 19.243656635284424
[2022-09-11 01:42:29,281] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:42:29,281] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:42:29,333] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:42:29,334] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:42:29,386] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:42:29,386] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:42:29,439] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:42:29,440] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:42:29,494] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:42:29,550] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:42:29,551] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:42:29,551] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:42:34,559] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:42:34,562] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:42:34,563] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:42:34,937] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:42:35,920] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:42:36,174] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:42:38,667] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:42:38,682] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 01:42:39,362] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:42:39,362] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:42:39,362] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:43:09,392] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:43:09,421] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:43:09,446] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:43:09,513] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.776584ms
[2022-09-11 01:43:09,513] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '71d2e168-c545-49b6-8fb9-a0f2404cf56e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.776584ms', 'executionTime': '64.625755ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:43:09,513] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:43:09,539] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:43:09,564] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:43:10,199] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 633.180041ms
[2022-09-11 01:43:10,199] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:43:10,273] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:43:10,315] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:43:10,323] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.472356ms
[2022-09-11 01:43:10,555] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:43:10,593] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:43:10,606] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:43:10,607] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:43:10,621] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:43:10,699] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:43:11,512] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed65f75bb95554be88470b3be1f366969job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:43:11,538] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed65f75bb95554be88470b3be1f366969job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:43:11,571] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.149024ms
[2022-09-11 01:43:11,571] - [base_gsi:282] INFO - BUILD INDEX on default(employeed65f75bb95554be88470b3be1f366969job_title) USING GSI
[2022-09-11 01:43:12,602] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed65f75bb95554be88470b3be1f366969job_title) USING GSI
[2022-09-11 01:43:12,628] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed65f75bb95554be88470b3be1f366969job_title%29+USING+GSI
[2022-09-11 01:43:12,657] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.260926ms
[2022-09-11 01:43:13,688] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed65f75bb95554be88470b3be1f366969job_title'
[2022-09-11 01:43:13,713] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed65f75bb95554be88470b3be1f366969job_title%27
[2022-09-11 01:43:13,722] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.554922ms
[2022-09-11 01:43:14,754] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed65f75bb95554be88470b3be1f366969job_title'
[2022-09-11 01:43:14,779] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed65f75bb95554be88470b3be1f366969job_title%27
[2022-09-11 01:43:14,788] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.65115ms
[2022-09-11 01:43:15,256] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 01:43:15,423] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:43:16,320] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:43:16,510] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-11 01:43:16,510] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:43:16,689] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-11 01:43:16,689] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-11 01:43:16,690] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2556869797, 0, b'')
[2022-09-11 01:43:16,790] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:43:16,818] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:43:16,844] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 01:43:16,849] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.407949ms
[2022-09-11 01:43:16,849] - [task:3246] INFO - {'requestID': '4e2952b4-b88e-4dff-b3cf-422685e109a1', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeed65f75bb95554be88470b3be1f366969job_title', 'index_id': '59fad7cbc0099cb5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.407949ms', 'executionTime': '3.344274ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 01:43:16,849] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:43:16,850] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:43:16,850] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:43:16,850] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:43:16,851] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 01:43:16,851] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:43:16,851] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:43:16,851] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:43:16,852] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:43:17,851] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:43:17,882] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:43:17,909] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 01:43:18,022] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 103.822556ms
[2022-09-11 01:43:18,022] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:43:18,022] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:43:18,821] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:43:18,822] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:43:19,852] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed65f75bb95554be88470b3be1f366969job_title'
[2022-09-11 01:43:19,880] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed65f75bb95554be88470b3be1f366969job_title%27
[2022-09-11 01:43:19,891] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.765393ms
[2022-09-11 01:43:19,917] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed65f75bb95554be88470b3be1f366969job_title ON default USING GSI
[2022-09-11 01:43:19,942] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed65f75bb95554be88470b3be1f366969job_title+ON+default+USING+GSI
[2022-09-11 01:43:19,978] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.202734ms
[2022-09-11 01:43:20,013] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed65f75bb95554be88470b3be1f366969job_title'
[2022-09-11 01:43:20,039] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed65f75bb95554be88470b3be1f366969job_title%27
[2022-09-11 01:43:20,047] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.787339ms
[2022-09-11 01:43:20,047] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '34d4e6df-337d-4924-b1f7-4fe91e92322c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.787339ms', 'executionTime': '5.715892ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:43:20,149] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:43:20,153] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:20,153] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:20,598] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:20,655] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:43:20,656] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:43:20,734] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:43:20,791] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:43:20,795] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:20,795] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:21,269] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:21,329] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:43:21,330] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:43:21,411] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:43:21,469] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:43:21,469] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:43:21,470] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:43:21,496] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:43:21,523] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:43:21,530] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.56145ms
[2022-09-11 01:43:21,558] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:43:21,585] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:43:21,640] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.33583ms
[2022-09-11 01:43:21,710] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:43:21,710] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.079483220755655, 'mem_free': 13027790848, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:43:21,710] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:43:21,715] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:21,715] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:22,191] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:22,196] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:22,196] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:22,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:22,833] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:22,833] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:23,605] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:23,612] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:23,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:24,389] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:29,221] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:43:29,366] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:43:29,718] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:43:29,747] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:43:29,747] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:43:29,803] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:43:29,857] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:43:29,909] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:43:29,910] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:43:29,988] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:43:29,989] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:43:30,016] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:43:30,146] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:43:30,146] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:43:30,172] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:43:30,198] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:43:30,198] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:43:30,224] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:43:30,249] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:43:30,249] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:43:30,275] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:43:30,301] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:43:30,301] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:43:30,327] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:43:30,327] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:43:30,328] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:43:30,328] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.395s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_5'}
[2022-09-11 01:43:30,429] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:30,429] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:30,862] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:30,892] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:43:30,975] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:43:30,976] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-11 01:43:30,976] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:43:31,455] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:43:31,483] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13110226944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:43:31,509] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:43:31,509] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:43:31,510] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:43:31,549] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:43:31,584] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:43:31,584] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:43:31,612] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:43:31,613] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:43:31,613] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:43:31,614] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:43:31,665] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:43:31,668] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:31,668] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:32,115] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:32,117] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:43:32,212] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:43:32,213] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:43:32,246] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:43:32,247] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:43:32,371] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:43:32,398] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13108379648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:43:32,424] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:43:32,451] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:43:32,452] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:43:32,504] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:43:32,508] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:32,508] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:32,918] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:32,919] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:43:33,013] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:43:33,014] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:43:33,043] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:43:33,044] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:43:33,158] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:43:33,187] - [task:165] INFO -  {'uptime': '397', 'memoryTotal': 15466930176, 'memoryFree': 13101121536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:43:33,212] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:43:33,240] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:43:33,240] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:43:33,291] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:43:33,295] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:33,295] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:33,721] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:33,723] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:43:33,814] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:43:33,815] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:43:33,845] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:43:33,845] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:43:33,967] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:43:33,995] - [task:165] INFO -  {'uptime': '396', 'memoryTotal': 15466930176, 'memoryFree': 13110161408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:43:34,021] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:43:34,048] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:43:34,048] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:43:34,098] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:43:34,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:34,102] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:34,505] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:34,506] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:43:34,596] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:43:34,598] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:43:34,626] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:43:34,626] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:43:34,722] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:43:35,268] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:43:40,273] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:43:40,358] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:43:40,363] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:40,363] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:40,795] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:40,797] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:43:40,887] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:43:40,888] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:43:40,888] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:43:41,836] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:43:41,892] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 01:43:41,892] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:43:42,783] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:43:43,100] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:43:43,380] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:43:43,384] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:43:43,446] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:43,446] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:43,881] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:43,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:43,887] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:44,291] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:44,296] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:44,296] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:44,975] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:44,983] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:44,984] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:45,742] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:50,612] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:43:50,612] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.47031631301624, 'mem_free': 13107634176, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:43:50,612] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:43:50,612] - [basetestcase:467] INFO - Time to execute basesetup : 20.18591570854187
[2022-09-11 01:43:50,664] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:43:50,664] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:43:50,719] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:43:50,719] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:43:50,772] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:43:50,772] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:43:50,824] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:43:50,824] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:43:50,876] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:43:50,937] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:43:50,937] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:43:50,938] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:43:55,948] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:43:55,952] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:43:55,952] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:43:56,399] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:43:57,398] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:43:57,647] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:44:00,127] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:44:00,143] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 01:44:01,225] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:44:01,226] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:44:01,226] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:44:31,243] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:44:31,272] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:44:31,297] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:44:31,364] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.189527ms
[2022-09-11 01:44:31,364] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '57632211-08ef-4d8e-b3e5-eff249da18d7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.189527ms', 'executionTime': '65.057018ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:44:31,365] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:44:31,391] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:44:31,417] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:44:32,062] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 643.307798ms
[2022-09-11 01:44:32,062] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:44:32,128] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:44:32,163] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:44:32,171] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.89082ms
[2022-09-11 01:44:32,369] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:44:32,405] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:44:32,428] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:44:32,429] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:44:32,446] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:44:32,516] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:44:33,331] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee9a1ce42166da4b4c9dbeb17a835cd916job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:44:33,356] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee9a1ce42166da4b4c9dbeb17a835cd916job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:44:33,393] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.922404ms
[2022-09-11 01:44:33,394] - [base_gsi:282] INFO - BUILD INDEX on default(employee9a1ce42166da4b4c9dbeb17a835cd916job_title) USING GSI
[2022-09-11 01:44:34,425] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee9a1ce42166da4b4c9dbeb17a835cd916job_title) USING GSI
[2022-09-11 01:44:34,450] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee9a1ce42166da4b4c9dbeb17a835cd916job_title%29+USING+GSI
[2022-09-11 01:44:34,482] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.046674ms
[2022-09-11 01:44:35,515] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9a1ce42166da4b4c9dbeb17a835cd916job_title'
[2022-09-11 01:44:35,541] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9a1ce42166da4b4c9dbeb17a835cd916job_title%27
[2022-09-11 01:44:35,549] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.170888ms
[2022-09-11 01:44:36,582] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9a1ce42166da4b4c9dbeb17a835cd916job_title'
[2022-09-11 01:44:36,608] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9a1ce42166da4b4c9dbeb17a835cd916job_title%27
[2022-09-11 01:44:36,615] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.592073ms
[2022-09-11 01:44:37,101] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:44:37,270] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:44:39,940] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:44:40,620] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:44:40,653] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:44:40,681] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 01:44:40,685] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.148588ms
[2022-09-11 01:44:40,685] - [task:3246] INFO - {'requestID': '44d7a1b8-6303-4b93-ac9b-92c8d477705d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee9a1ce42166da4b4c9dbeb17a835cd916job_title', 'index_id': '8fb8d96631b2b454', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.148588ms', 'executionTime': '2.073212ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 01:44:40,685] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:44:40,686] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:44:40,686] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:44:40,686] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:44:40,687] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 01:44:40,687] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:44:40,687] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:44:40,687] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:44:40,688] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:44:41,687] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:44:41,716] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:44:41,742] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 01:44:41,934] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 175.853709ms
[2022-09-11 01:44:41,934] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:44:41,935] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:44:43,529] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:44:43,529] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:44:44,560] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9a1ce42166da4b4c9dbeb17a835cd916job_title'
[2022-09-11 01:44:44,585] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9a1ce42166da4b4c9dbeb17a835cd916job_title%27
[2022-09-11 01:44:44,594] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.389621ms
[2022-09-11 01:44:44,620] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee9a1ce42166da4b4c9dbeb17a835cd916job_title ON default USING GSI
[2022-09-11 01:44:44,646] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee9a1ce42166da4b4c9dbeb17a835cd916job_title+ON+default+USING+GSI
[2022-09-11 01:44:44,697] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.928483ms
[2022-09-11 01:44:44,732] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9a1ce42166da4b4c9dbeb17a835cd916job_title'
[2022-09-11 01:44:44,759] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9a1ce42166da4b4c9dbeb17a835cd916job_title%27
[2022-09-11 01:44:44,766] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.635795ms
[2022-09-11 01:44:44,766] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '50c4a093-721a-482e-a421-17867c567da2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.635795ms', 'executionTime': '5.564394ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:44:44,873] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:44:44,877] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:44,877] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:45,439] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:45,497] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:44:45,497] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:44:45,586] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:44:45,641] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:44:45,644] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:45,644] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:46,192] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:46,251] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:44:46,252] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:44:46,343] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:44:46,399] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:44:46,400] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:44:46,400] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:44:46,426] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:44:46,451] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:44:46,459] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.902662ms
[2022-09-11 01:44:46,485] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:44:46,511] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:44:46,560] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.259394ms
[2022-09-11 01:44:46,637] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:44:46,638] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.29205504877747, 'mem_free': 12930760704, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:44:46,638] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:44:46,642] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:46,642] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:47,189] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:47,194] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:47,194] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:47,731] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:47,735] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:47,735] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:48,628] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:48,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:48,640] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:49,577] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:55,367] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:44:55,508] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:44:56,447] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:44:56,474] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:44:56,475] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:44:56,529] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:44:56,582] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:44:56,644] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:44:56,645] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:44:56,729] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:44:56,730] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:44:56,755] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:44:56,883] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:44:56,884] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:44:56,910] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:44:56,936] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:44:56,936] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:44:56,963] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:44:56,988] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:44:56,988] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:44:57,014] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:44:57,040] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:44:57,040] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:44:57,066] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:44:57,066] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:44:57,067] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:44:57,067] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 86.692s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_6'}
[2022-09-11 01:44:57,175] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:57,175] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:57,656] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:57,689] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:44:57,768] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:44:57,769] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-11 01:44:57,769] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:44:58,202] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:44:58,230] - [task:165] INFO -  {'uptime': '489', 'memoryTotal': 15466930176, 'memoryFree': 12947968000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:44:58,257] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:44:58,257] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:44:58,257] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:44:58,293] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:44:58,341] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:44:58,341] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:44:58,371] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:44:58,372] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:44:58,372] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:44:58,373] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:44:58,421] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:44:58,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:58,425] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:58,947] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:58,948] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:44:59,048] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:44:59,050] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:44:59,083] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:44:59,084] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:44:59,211] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:44:59,239] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13053210624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:44:59,265] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:44:59,294] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:44:59,294] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:44:59,347] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:44:59,350] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:44:59,350] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:44:59,873] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:44:59,875] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:44:59,978] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:44:59,980] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:45:00,008] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:45:00,009] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:45:00,135] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:45:00,161] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13053104128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:45:00,187] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:45:00,216] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:45:00,216] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:45:00,267] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:45:00,271] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:00,271] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:00,775] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:00,776] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:45:00,874] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:45:00,875] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:45:00,905] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:45:00,905] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:45:01,024] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:45:01,052] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13053472768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:45:01,078] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:45:01,107] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:45:01,108] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:45:01,163] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:45:01,166] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:01,166] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:01,722] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:01,723] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:45:01,834] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:45:01,836] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:45:01,867] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:45:01,867] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:45:01,962] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:45:02,515] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:45:07,517] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:45:07,600] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:45:07,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:07,606] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:08,128] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:08,129] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:45:08,228] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:45:08,229] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:45:08,230] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:45:09,080] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:45:09,133] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 01:45:09,134] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:45:10,062] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:45:10,376] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:45:10,653] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:45:10,657] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:45:10,714] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:10,714] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:11,246] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:11,251] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:11,251] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:11,848] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:11,856] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:11,856] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:12,763] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:12,771] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:12,771] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:13,655] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:19,065] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:45:19,065] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.56510418802063, 'mem_free': 13055262720, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:45:19,065] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:45:19,066] - [basetestcase:467] INFO - Time to execute basesetup : 21.892688751220703
[2022-09-11 01:45:19,117] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:45:19,117] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:45:19,174] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:45:19,174] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:45:19,227] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:45:19,227] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:45:19,279] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:45:19,279] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:45:19,330] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:45:19,390] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:45:19,391] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:45:19,391] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:45:24,403] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:45:24,406] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:45:24,407] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:45:24,935] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:45:25,964] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:45:26,228] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:45:28,490] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:45:28,506] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 01:45:29,901] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:45:29,901] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:45:29,902] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:45:59,927] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:45:59,957] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:45:59,983] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:46:00,054] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.794916ms
[2022-09-11 01:46:00,054] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9ae1ea17-c8be-4850-890c-cdcdeb4953be', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.794916ms', 'executionTime': '68.722088ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:46:00,055] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:46:00,082] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:46:00,108] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:46:00,772] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 662.260733ms
[2022-09-11 01:46:00,773] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:46:00,839] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:46:00,869] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:46:00,878] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.023188ms
[2022-09-11 01:46:01,078] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:46:01,117] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:46:01,132] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:46:01,132] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:46:01,145] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:46:01,229] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:46:02,043] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0fe99a3ad7b7408190497d9f6a0afac7job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:46:02,069] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0fe99a3ad7b7408190497d9f6a0afac7job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:46:02,102] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.281292ms
[2022-09-11 01:46:02,103] - [base_gsi:282] INFO - BUILD INDEX on default(employee0fe99a3ad7b7408190497d9f6a0afac7job_title) USING GSI
[2022-09-11 01:46:03,132] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee0fe99a3ad7b7408190497d9f6a0afac7job_title) USING GSI
[2022-09-11 01:46:03,158] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee0fe99a3ad7b7408190497d9f6a0afac7job_title%29+USING+GSI
[2022-09-11 01:46:03,183] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.275118ms
[2022-09-11 01:46:04,218] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0fe99a3ad7b7408190497d9f6a0afac7job_title'
[2022-09-11 01:46:04,244] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fe99a3ad7b7408190497d9f6a0afac7job_title%27
[2022-09-11 01:46:04,253] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.433573ms
[2022-09-11 01:46:05,284] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0fe99a3ad7b7408190497d9f6a0afac7job_title'
[2022-09-11 01:46:05,311] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fe99a3ad7b7408190497d9f6a0afac7job_title%27
[2022-09-11 01:46:05,322] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.295969ms
[2022-09-11 01:46:05,832] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 01:46:06,003] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:46:07,129] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:46:07,748] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 01:46:07,920] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:46:09,221] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:46:09,615] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:46:09,790] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:46:13,426] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:46:14,341] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:46:14,373] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:46:14,399] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 01:46:14,403] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.891792ms
[2022-09-11 01:46:14,404] - [task:3246] INFO - {'requestID': '7ca78b57-5609-4e0e-b088-dfe0890151b5', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0fe99a3ad7b7408190497d9f6a0afac7job_title', 'index_id': '43c4947450ac9c0b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.891792ms', 'executionTime': '2.830493ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 01:46:14,404] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:46:14,404] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:46:14,405] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:46:14,405] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:46:14,406] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 01:46:14,406] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:46:14,406] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:46:14,406] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:46:14,407] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 01:46:15,405] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:46:15,435] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 01:46:15,460] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 01:46:15,867] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 184.547492ms
[2022-09-11 01:46:15,867] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:46:15,868] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:46:17,474] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:46:17,474] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 01:46:18,504] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0fe99a3ad7b7408190497d9f6a0afac7job_title'
[2022-09-11 01:46:18,530] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fe99a3ad7b7408190497d9f6a0afac7job_title%27
[2022-09-11 01:46:18,539] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.679822ms
[2022-09-11 01:46:18,564] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0fe99a3ad7b7408190497d9f6a0afac7job_title ON default USING GSI
[2022-09-11 01:46:18,589] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0fe99a3ad7b7408190497d9f6a0afac7job_title+ON+default+USING+GSI
[2022-09-11 01:46:18,630] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.062008ms
[2022-09-11 01:46:18,664] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0fe99a3ad7b7408190497d9f6a0afac7job_title'
[2022-09-11 01:46:18,689] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fe99a3ad7b7408190497d9f6a0afac7job_title%27
[2022-09-11 01:46:18,696] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.462865ms
[2022-09-11 01:46:18,696] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '026a0338-2608-43e6-95a7-2529ba32a6d3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.462865ms', 'executionTime': '5.406622ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:46:18,798] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:46:18,801] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:18,801] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:19,440] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:19,494] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:46:19,494] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:46:19,597] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:46:19,651] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:46:19,654] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:19,655] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:20,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:20,341] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:46:20,341] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:46:20,458] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:46:20,512] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:46:20,513] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:46:20,513] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:46:20,538] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:46:20,563] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:46:20,573] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.090194ms
[2022-09-11 01:46:20,598] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:46:20,623] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:46:20,680] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.378736ms
[2022-09-11 01:46:20,744] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:46:20,744] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.91366888705651, 'mem_free': 12911697920, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:46:20,744] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:46:20,748] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:20,748] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:21,389] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:21,393] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:21,393] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:21,998] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:22,004] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:22,004] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:22,822] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:22,830] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:22,830] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:23,965] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:31,518] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:46:31,658] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:46:32,672] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:46:32,701] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:46:32,701] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:46:32,756] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:46:32,822] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:46:32,874] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:46:32,874] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:46:32,954] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:46:32,955] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:46:32,981] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:46:33,117] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:46:33,118] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:46:33,145] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:46:33,171] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:46:33,172] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:46:33,199] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:46:33,225] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:46:33,225] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:46:33,252] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:46:33,279] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:46:33,279] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:46:33,307] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:46:33,307] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 01:46:33,307] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:46:33,308] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 96.190s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_7'}
[2022-09-11 01:46:33,444] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:33,444] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:34,076] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:34,110] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:46:34,194] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:46:34,194] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-11 01:46:34,195] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:46:34,471] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:46:34,498] - [task:165] INFO -  {'uptime': '585', 'memoryTotal': 15466930176, 'memoryFree': 12990734336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:46:34,523] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:46:34,523] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:46:34,523] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:46:34,554] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:46:34,597] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:46:34,597] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:46:34,626] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:46:34,626] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:46:34,627] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:46:34,627] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:46:34,675] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:46:34,678] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:34,678] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:35,263] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:35,264] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:46:35,379] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:46:35,380] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:46:35,410] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:46:35,411] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:46:35,549] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:46:35,575] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 12988145664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:46:35,600] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:46:35,627] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:46:35,627] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:46:35,678] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:46:35,681] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:35,681] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:36,250] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:36,251] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:46:36,363] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:46:36,364] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:46:36,395] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:46:36,396] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:46:36,512] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:46:36,539] - [task:165] INFO -  {'uptime': '583', 'memoryTotal': 15466930176, 'memoryFree': 12980842496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:46:36,566] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:46:36,594] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:46:36,594] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:46:36,646] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:46:36,651] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:36,651] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:37,272] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:37,273] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:46:37,385] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:46:37,386] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:46:37,415] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:46:37,416] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:46:37,532] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:46:37,558] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 13005299712, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:46:37,584] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:46:37,611] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:46:37,611] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:46:37,662] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:46:37,664] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:37,665] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:38,269] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:38,270] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:46:38,383] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:46:38,384] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:46:38,414] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:46:38,415] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:46:38,508] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:46:39,140] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:46:44,145] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:46:44,229] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:46:44,233] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:44,233] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:44,816] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:44,817] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:46:44,930] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:46:44,931] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:46:44,932] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:46:45,667] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:46:45,723] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 01:46:45,723] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:46:46,608] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:46:46,916] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:46:47,203] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:46:47,207] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 01:46:47,268] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:47,268] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:47,874] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:47,878] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:47,879] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:48,629] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:48,641] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:48,641] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:49,656] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:49,669] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:46:49,669] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:46:50,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:46:56,947] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:46:56,947] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.23106665127754, 'mem_free': 12997255168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:46:56,947] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:46:56,947] - [basetestcase:467] INFO - Time to execute basesetup : 23.505675554275513
[2022-09-11 01:46:56,998] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:46:56,998] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:46:57,051] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:46:57,051] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:46:57,104] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:46:57,104] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:46:57,165] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:46:57,166] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:46:57,216] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:46:57,272] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:46:57,273] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:46:57,273] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:47:02,282] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:47:02,285] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:02,286] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:02,934] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:03,990] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:47:04,168] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:47:06,389] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:47:06,472] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:47:06,472] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:47:06,472] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:47:36,503] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:47:36,532] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:47:36,558] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:47:36,624] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.78829ms
[2022-09-11 01:47:36,625] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '016c4f7d-226d-4e7b-96b1-46172035b874', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.78829ms', 'executionTime': '64.719521ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:47:36,625] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:47:36,650] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:47:36,676] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:47:37,320] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 641.834942ms
[2022-09-11 01:47:37,320] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:47:37,387] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:47:37,444] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:47:37,456] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.946579ms
[2022-09-11 01:47:37,673] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:47:37,706] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:47:37,726] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:47:37,727] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:47:37,751] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:47:37,752] - [base_gsi:326] INFO - []
[2022-09-11 01:47:38,637] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef3b399a196c344da87d44c322de571f0job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:47:38,662] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef3b399a196c344da87d44c322de571f0job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:47:38,696] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.167326ms
[2022-09-11 01:47:38,731] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef3b399a196c344da87d44c322de571f0join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:47:38,756] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef3b399a196c344da87d44c322de571f0join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:47:38,788] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.817984ms
[2022-09-11 01:47:38,789] - [base_gsi:282] INFO - BUILD INDEX on default(employeef3b399a196c344da87d44c322de571f0job_title,employeef3b399a196c344da87d44c322de571f0join_yr) USING GSI
[2022-09-11 01:47:39,820] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeef3b399a196c344da87d44c322de571f0job_title,employeef3b399a196c344da87d44c322de571f0join_yr) USING GSI
[2022-09-11 01:47:39,846] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeef3b399a196c344da87d44c322de571f0job_title%2Cemployeef3b399a196c344da87d44c322de571f0join_yr%29+USING+GSI
[2022-09-11 01:47:39,887] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.024892ms
[2022-09-11 01:47:40,918] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef3b399a196c344da87d44c322de571f0job_title'
[2022-09-11 01:47:40,944] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef3b399a196c344da87d44c322de571f0job_title%27
[2022-09-11 01:47:40,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.294177ms
[2022-09-11 01:47:41,986] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef3b399a196c344da87d44c322de571f0job_title'
[2022-09-11 01:47:42,012] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef3b399a196c344da87d44c322de571f0job_title%27
[2022-09-11 01:47:42,019] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.027575ms
[2022-09-11 01:47:42,045] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef3b399a196c344da87d44c322de571f0join_yr'
[2022-09-11 01:47:42,070] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef3b399a196c344da87d44c322de571f0join_yr%27
[2022-09-11 01:47:42,072] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.365413ms
[2022-09-11 01:47:43,103] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef3b399a196c344da87d44c322de571f0job_title'
[2022-09-11 01:47:43,128] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef3b399a196c344da87d44c322de571f0job_title%27
[2022-09-11 01:47:43,138] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.692617ms
[2022-09-11 01:47:43,163] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef3b399a196c344da87d44c322de571f0job_title ON default USING GSI
[2022-09-11 01:47:43,188] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeef3b399a196c344da87d44c322de571f0job_title+ON+default+USING+GSI
[2022-09-11 01:47:43,225] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.541468ms
[2022-09-11 01:47:43,265] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef3b399a196c344da87d44c322de571f0join_yr'
[2022-09-11 01:47:43,290] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef3b399a196c344da87d44c322de571f0join_yr%27
[2022-09-11 01:47:43,298] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.455072ms
[2022-09-11 01:47:43,325] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef3b399a196c344da87d44c322de571f0join_yr ON default USING GSI
[2022-09-11 01:47:43,351] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeef3b399a196c344da87d44c322de571f0join_yr+ON+default+USING+GSI
[2022-09-11 01:47:43,390] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.595024ms
[2022-09-11 01:47:43,425] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef3b399a196c344da87d44c322de571f0job_title'
[2022-09-11 01:47:43,451] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef3b399a196c344da87d44c322de571f0job_title%27
[2022-09-11 01:47:43,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.155237ms
[2022-09-11 01:47:43,458] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2bb4148f-c416-4478-8632-3ebe4f754c25', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.155237ms', 'executionTime': '5.0891ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:47:43,483] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef3b399a196c344da87d44c322de571f0join_yr'
[2022-09-11 01:47:43,508] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef3b399a196c344da87d44c322de571f0join_yr%27
[2022-09-11 01:47:43,511] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.159095ms
[2022-09-11 01:47:43,511] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '837458cc-bc8f-4a13-9c18-9f96659d608a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.159095ms', 'executionTime': '1.112514ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:47:43,611] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:47:43,614] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:43,615] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:44,248] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:44,307] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:47:44,308] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:47:44,409] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:47:44,467] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:47:44,470] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:44,471] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:45,073] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:45,128] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:47:45,129] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:47:45,227] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:47:45,283] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:47:45,284] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:47:45,284] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:47:45,309] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:47:45,335] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:47:45,345] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.015305ms
[2022-09-11 01:47:45,371] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:47:45,397] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:47:45,445] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.932668ms
[2022-09-11 01:47:45,516] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:47:45,517] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.542344713087724, 'mem_free': 12881543168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:47:45,517] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:47:45,522] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:45,522] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:46,103] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:46,108] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:46,109] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:46,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:46,698] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:46,698] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:47,507] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:47,516] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:47,516] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:48,527] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:55,474] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-11 01:47:55,622] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:47:56,192] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:47:56,221] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:47:56,221] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:47:56,281] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:47:56,338] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:47:56,398] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:47:56,399] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:47:56,483] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:47:56,484] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:47:56,510] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:47:56,636] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:47:56,636] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:47:56,662] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:47:56,688] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:47:56,688] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:47:56,715] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:47:56,746] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:47:56,746] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:47:56,777] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:47:56,803] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:47:56,803] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:47:56,830] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:47:56,830] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 01:47:56,830] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:47:56,831] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 83.440s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_8'}
[2022-09-11 01:47:56,908] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:56,908] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:57,511] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:57,543] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:47:57,621] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:47:57,621] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-11 01:47:57,622] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:47:57,934] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:47:57,960] - [task:165] INFO -  {'uptime': '669', 'memoryTotal': 15466930176, 'memoryFree': 13005864960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:47:57,986] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:47:57,987] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:47:57,987] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:47:58,019] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:47:58,053] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:47:58,054] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:47:58,081] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:47:58,082] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:47:58,082] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:47:58,082] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:47:58,129] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:47:58,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:58,132] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:58,713] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:58,714] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:47:58,826] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:47:58,827] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:47:58,858] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:47:58,858] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:47:58,979] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:47:59,005] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13005398016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:47:59,031] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:47:59,062] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:47:59,062] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:47:59,112] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:47:59,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:47:59,115] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:47:59,688] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:47:59,689] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:47:59,810] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:47:59,811] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:47:59,842] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:47:59,843] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:47:59,961] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:47:59,987] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13005643776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:48:00,014] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:48:00,044] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:48:00,044] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:48:00,097] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:48:00,102] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:00,102] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:00,726] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:00,727] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:48:00,854] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:48:00,855] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:48:00,888] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:48:00,889] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:48:01,012] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:48:01,040] - [task:165] INFO -  {'uptime': '667', 'memoryTotal': 15466930176, 'memoryFree': 13005611008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:48:01,067] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:48:01,095] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:48:01,096] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:48:01,148] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:48:01,151] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:01,152] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:01,785] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:01,786] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:48:01,913] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:48:01,914] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:48:01,945] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:48:01,946] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:48:02,040] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:48:02,703] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:48:07,709] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:48:07,795] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:48:07,799] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:07,799] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:08,415] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:08,416] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:48:08,531] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:48:08,532] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:48:08,532] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:48:09,170] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:48:09,224] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 01:48:09,224] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:48:10,286] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:48:10,608] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:48:10,914] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:48:10,918] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 01:48:10,974] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:10,974] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:11,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:11,596] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:11,596] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:12,500] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:12,512] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:12,513] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:13,568] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:13,579] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:13,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:14,652] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:20,933] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:48:20,934] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.68058003998875, 'mem_free': 12998438912, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:48:20,934] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:48:20,934] - [basetestcase:467] INFO - Time to execute basesetup : 24.0289089679718
[2022-09-11 01:48:20,990] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:48:20,990] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:48:21,044] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:48:21,044] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:48:21,098] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:48:21,098] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:48:21,151] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:48:21,152] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:48:21,203] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:48:21,261] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:48:21,261] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:48:21,261] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:48:26,270] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:48:26,274] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:48:26,274] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:48:26,907] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:48:27,955] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:48:28,230] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:48:30,367] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:48:30,446] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:48:30,446] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:48:30,447] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:49:00,461] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:49:00,491] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:49:00,517] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:49:00,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.171978ms
[2022-09-11 01:49:00,587] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '18320677-b248-4982-a9d0-5f5fbfee2f24', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.171978ms', 'executionTime': '67.103131ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:49:00,587] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:49:00,613] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:49:00,638] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:49:01,303] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 663.020818ms
[2022-09-11 01:49:01,303] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:49:01,390] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:49:01,432] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:49:01,443] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.557269ms
[2022-09-11 01:49:01,666] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:49:01,702] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:49:01,726] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:49:01,727] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:49:01,737] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:49:01,738] - [base_gsi:326] INFO - []
[2022-09-11 01:49:02,623] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:49:02,648] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:49:02,679] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.421961ms
[2022-09-11 01:49:02,680] - [base_gsi:282] INFO - BUILD INDEX on default(employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr) USING GSI
[2022-09-11 01:49:03,711] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr) USING GSI
[2022-09-11 01:49:03,736] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr%29+USING+GSI
[2022-09-11 01:49:03,762] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.07571ms
[2022-09-11 01:49:04,793] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr'
[2022-09-11 01:49:04,818] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr%27
[2022-09-11 01:49:04,828] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.550959ms
[2022-09-11 01:49:05,858] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr'
[2022-09-11 01:49:05,884] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr%27
[2022-09-11 01:49:05,893] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.458159ms
[2022-09-11 01:49:06,925] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr'
[2022-09-11 01:49:06,953] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr%27
[2022-09-11 01:49:06,965] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.946967ms
[2022-09-11 01:49:06,995] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr ON default USING GSI
[2022-09-11 01:49:07,020] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr+ON+default+USING+GSI
[2022-09-11 01:49:07,066] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.0828ms
[2022-09-11 01:49:07,100] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr'
[2022-09-11 01:49:07,125] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee79fb0d32eb524379bc1a1cfc650adb32job_title_join_yr%27
[2022-09-11 01:49:07,133] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.99165ms
[2022-09-11 01:49:07,133] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3ee93864-5da2-4efe-b8e3-2a989a519eac', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.99165ms', 'executionTime': '5.920316ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:49:07,237] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:49:07,241] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:07,241] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:07,899] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:07,956] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:49:07,957] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:49:08,062] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:49:08,122] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:49:08,125] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:08,125] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:08,787] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:08,845] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:49:08,845] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:49:08,952] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:49:09,012] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:49:09,012] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:49:09,012] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:49:09,039] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:49:09,067] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:49:09,078] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.874835ms
[2022-09-11 01:49:09,107] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:49:09,137] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:49:09,177] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.380072ms
[2022-09-11 01:49:09,257] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:49:09,258] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.964598051605935, 'mem_free': 12891971584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:49:09,258] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:49:09,262] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:09,263] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:09,912] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:09,918] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:09,918] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:11,001] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:11,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:11,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:12,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:12,117] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:12,118] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:13,199] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:19,302] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-11 01:49:19,445] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:49:19,816] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:49:19,845] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:49:19,846] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:49:19,903] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:49:19,956] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:49:20,006] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:49:20,007] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:49:20,090] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:49:20,091] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:49:20,117] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:49:20,245] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:49:20,245] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:49:20,273] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:49:20,299] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:49:20,300] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:49:20,326] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:49:20,352] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:49:20,353] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:49:20,380] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:49:20,405] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:49:20,406] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:49:20,432] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:49:20,432] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 01:49:20,432] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:49:20,433] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 83.579s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_9'}
[2022-09-11 01:49:20,518] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:20,518] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:21,125] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:21,158] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:49:21,240] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:49:21,240] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-11 01:49:21,241] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:49:21,543] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:49:21,569] - [task:165] INFO -  {'uptime': '752', 'memoryTotal': 15466930176, 'memoryFree': 13008302080, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:49:21,595] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:49:21,595] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:49:21,595] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:49:21,632] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:49:21,667] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:49:21,668] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:49:21,696] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:49:21,697] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:49:21,697] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:49:21,697] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:49:21,745] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:49:21,748] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:21,749] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:22,386] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:22,387] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:49:22,547] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:49:22,548] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:49:22,578] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:49:22,579] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:49:22,699] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:49:22,725] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 13007552512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:49:22,753] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:49:22,782] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:49:22,782] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:49:22,836] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:49:22,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:22,841] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:23,444] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:23,445] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:49:23,561] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:49:23,562] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:49:23,591] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:49:23,592] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:49:23,706] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:49:23,732] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 13006032896, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:49:23,758] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:49:23,785] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:49:23,785] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:49:23,837] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:49:23,841] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:23,842] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:24,431] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:24,433] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:49:24,552] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:49:24,553] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:49:24,582] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:49:24,583] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:49:24,700] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:49:24,728] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 13007220736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:49:24,753] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:49:24,780] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:49:24,780] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:49:24,832] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:49:24,836] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:24,837] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:25,438] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:25,439] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:49:25,553] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:49:25,554] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:49:25,583] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:49:25,584] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:49:25,675] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:49:26,216] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:49:31,218] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:49:31,306] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:49:31,309] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:31,310] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:31,906] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:31,907] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:49:32,023] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:49:32,024] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:49:32,025] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:49:32,837] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:49:32,894] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 01:49:32,895] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:49:33,815] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:49:34,131] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:49:34,419] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:49:34,423] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 01:49:34,484] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:34,484] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:35,116] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:35,121] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:35,121] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:35,977] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:35,985] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:35,985] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:37,088] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:37,098] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:37,098] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:38,159] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:44,523] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:49:44,523] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.05806610240314, 'mem_free': 13003243520, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:49:44,524] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:49:44,524] - [basetestcase:467] INFO - Time to execute basesetup : 24.00893807411194
[2022-09-11 01:49:44,578] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:49:44,578] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:49:44,631] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:49:44,632] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:49:44,686] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:49:44,686] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:49:44,738] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:49:44,738] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:49:44,788] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:49:44,848] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:49:44,849] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:49:44,849] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:49:49,859] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:49:49,863] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:49:49,863] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:49:50,504] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:49:51,657] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:49:51,822] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:49:53,984] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:49:54,060] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:49:54,060] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:49:54,060] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:50:24,087] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:50:24,117] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:50:24,143] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:50:24,210] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.791654ms
[2022-09-11 01:50:24,210] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a57ea73c-5323-403a-a6e7-396823a80584', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.791654ms', 'executionTime': '65.726247ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:50:24,211] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:50:24,236] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:50:24,261] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:50:24,930] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 666.779401ms
[2022-09-11 01:50:24,930] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:50:24,992] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:50:25,029] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:50:25,037] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.214249ms
[2022-09-11 01:50:25,239] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:50:25,272] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:50:25,306] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:50:25,306] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:50:25,329] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:50:25,393] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:50:26,205] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeade49fe0e02941f5966e08b451baa60ejoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:50:26,236] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeade49fe0e02941f5966e08b451baa60ejoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:50:26,270] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.257503ms
[2022-09-11 01:50:26,270] - [base_gsi:282] INFO - BUILD INDEX on default(`employeeade49fe0e02941f5966e08b451baa60ejoin_yr`) USING GSI
[2022-09-11 01:50:27,301] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeeade49fe0e02941f5966e08b451baa60ejoin_yr`) USING GSI
[2022-09-11 01:50:27,326] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeeade49fe0e02941f5966e08b451baa60ejoin_yr%60%29+USING+GSI
[2022-09-11 01:50:27,346] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.455682ms
[2022-09-11 01:50:27,403] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeade49fe0e02941f5966e08b451baa60ejoin_yr'
[2022-09-11 01:50:27,433] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeade49fe0e02941f5966e08b451baa60ejoin_yr%27
[2022-09-11 01:50:27,442] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.3322ms
[2022-09-11 01:50:28,473] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeade49fe0e02941f5966e08b451baa60ejoin_yr'
[2022-09-11 01:50:28,498] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeade49fe0e02941f5966e08b451baa60ejoin_yr%27
[2022-09-11 01:50:28,507] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.424149ms
[2022-09-11 01:50:29,537] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeade49fe0e02941f5966e08b451baa60ejoin_yr'
[2022-09-11 01:50:29,563] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeade49fe0e02941f5966e08b451baa60ejoin_yr%27
[2022-09-11 01:50:29,572] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.516893ms
[2022-09-11 01:50:30,603] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:50:30,628] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 01:50:30,632] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.382842ms
[2022-09-11 01:50:30,632] - [base_gsi:504] INFO - {'requestID': '202eefde-21ea-4235-b5e4-e3f333664d6b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeade49fe0e02941f5966e08b451baa60ejoin_yr', 'index_id': '90478e3e8dcc62cb', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.382842ms', 'executionTime': '2.323436ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 01:50:30,633] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:50:30,633] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:50:30,633] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:50:30,633] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:50:30,635] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:50:30,635] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:50:30,635] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:50:30,636] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:50:30,696] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:50:30,735] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:50:30,761] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 01:50:30,918] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 144.269684ms
[2022-09-11 01:50:30,918] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:50:30,919] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:50:33,375] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:50:33,412] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeade49fe0e02941f5966e08b451baa60ejoin_yr'
[2022-09-11 01:50:33,441] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeade49fe0e02941f5966e08b451baa60ejoin_yr%27
[2022-09-11 01:50:33,448] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.71978ms
[2022-09-11 01:50:33,448] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b94f6ca0-21d4-46d3-955a-44ff296f4891', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.71978ms', 'executionTime': '4.647458ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:50:33,549] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:50:33,552] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:33,552] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:34,237] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:34,292] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:50:34,292] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:50:34,398] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:50:34,462] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:50:34,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:34,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:35,121] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:35,178] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:50:35,179] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:50:35,294] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:50:35,353] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:50:35,353] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:50:35,354] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:50:35,383] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:50:35,411] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:50:35,416] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.490051ms
[2022-09-11 01:50:35,416] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2fe5fd44-159d-4396-8d11-e79943046dfa', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.490051ms', 'executionTime': '3.423198ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:50:35,472] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:50:35,473] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.04106373144242, 'mem_free': 12843085824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:50:35,473] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:50:35,476] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:35,477] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:36,161] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:36,166] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:36,166] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:37,109] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:37,117] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:37,117] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:38,312] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:38,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:38,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:39,503] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:46,345] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-11 01:50:46,477] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:50:46,529] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:50:46,586] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:50:46,637] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:50:46,638] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:50:46,715] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:50:46,716] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:50:46,742] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:50:46,869] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:50:46,869] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:50:46,895] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:50:46,921] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:50:46,921] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:50:46,947] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:50:46,973] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:50:46,973] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:50:46,999] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:50:47,023] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:50:47,024] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:50:47,050] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:50:47,050] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 01:50:47,050] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:50:47,051] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.587s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_10'}
[2022-09-11 01:50:47,131] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:47,131] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:47,766] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:47,798] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:50:47,877] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:50:47,877] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-11 01:50:47,878] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:50:48,156] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:50:48,182] - [task:165] INFO -  {'uptime': '836', 'memoryTotal': 15466930176, 'memoryFree': 12987920384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:50:48,211] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:50:48,212] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:50:48,212] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:50:48,244] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:50:48,276] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:50:48,277] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:50:48,304] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:50:48,304] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:50:48,305] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:50:48,305] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:50:48,354] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:50:48,356] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:48,357] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:48,990] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:48,991] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:50:49,107] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:50:49,108] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:50:49,137] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:50:49,138] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:50:49,257] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:50:49,282] - [task:165] INFO -  {'uptime': '838', 'memoryTotal': 15466930176, 'memoryFree': 12987428864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:50:49,308] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:50:49,336] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:50:49,336] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:50:49,386] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:50:49,391] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:49,392] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:50,020] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:50,021] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:50:50,139] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:50:50,140] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:50:50,169] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:50:50,170] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:50:50,284] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:50:50,311] - [task:165] INFO -  {'uptime': '834', 'memoryTotal': 15466930176, 'memoryFree': 12986793984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:50:50,336] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:50:50,364] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:50:50,364] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:50:50,415] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:50:50,420] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:50,420] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:51,047] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:51,048] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:50:51,175] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:50:51,176] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:50:51,207] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:50:51,208] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:50:51,326] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:50:51,352] - [task:165] INFO -  {'uptime': '833', 'memoryTotal': 15466930176, 'memoryFree': 12986716160, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:50:51,377] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:50:51,404] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:50:51,405] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:50:51,456] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:50:51,459] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:51,459] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:52,079] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:52,080] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:50:52,197] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:50:52,198] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:50:52,228] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:50:52,229] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:50:52,317] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:50:52,856] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:50:57,861] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:50:57,945] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:50:57,954] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:50:57,954] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:50:58,608] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:50:58,609] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:50:58,729] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:50:58,730] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:50:58,730] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:50:59,430] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:50:59,486] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 01:50:59,486] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:51:00,430] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:51:00,766] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:51:01,086] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:51:01,089] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 01:51:01,157] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:51:01,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:51:01,839] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:51:01,844] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:51:01,844] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:51:02,874] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:51:02,882] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:51:02,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:51:03,992] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:51:04,001] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:51:04,001] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:51:05,113] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:51:11,868] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:51:11,868] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.02440859511828, 'mem_free': 12983525376, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:51:11,869] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:51:11,869] - [basetestcase:467] INFO - Time to execute basesetup : 24.74207305908203
[2022-09-11 01:51:11,919] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:51:11,919] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:51:11,972] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:51:11,972] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:51:12,026] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:51:12,026] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:51:12,077] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:51:12,078] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:51:12,128] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:51:12,186] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:51:12,186] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:51:12,187] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:51:17,200] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:51:17,203] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:51:17,204] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:51:17,831] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:51:18,890] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:51:19,180] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:51:21,292] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:51:21,369] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:51:21,369] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:51:21,369] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:51:51,400] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:51:51,429] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:51:51,454] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:51:51,520] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.107369ms
[2022-09-11 01:51:51,520] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c5f75041-5cd7-477c-99de-808cda0ea6ee', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.107369ms', 'executionTime': '64.03291ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:51:51,520] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:51:51,546] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:51:51,570] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:51:52,239] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 667.904791ms
[2022-09-11 01:51:52,240] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:51:52,303] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:51:52,356] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:51:52,364] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.755375ms
[2022-09-11 01:51:52,585] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:51:52,620] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:51:52,639] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:51:52,640] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:51:52,664] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:51:52,725] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:51:53,547] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeaf5c70879eda434884315cb2a06c9b3djoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:51:53,572] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeaf5c70879eda434884315cb2a06c9b3djoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:51:53,613] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.631884ms
[2022-09-11 01:51:53,614] - [base_gsi:282] INFO - BUILD INDEX on default(`employeeaf5c70879eda434884315cb2a06c9b3djoin_yr`) USING GSI
[2022-09-11 01:51:54,643] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeeaf5c70879eda434884315cb2a06c9b3djoin_yr`) USING GSI
[2022-09-11 01:51:54,671] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeeaf5c70879eda434884315cb2a06c9b3djoin_yr%60%29+USING+GSI
[2022-09-11 01:51:54,698] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.043879ms
[2022-09-11 01:51:54,741] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaf5c70879eda434884315cb2a06c9b3djoin_yr'
[2022-09-11 01:51:54,777] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaf5c70879eda434884315cb2a06c9b3djoin_yr%27
[2022-09-11 01:51:54,786] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.851845ms
[2022-09-11 01:51:55,816] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaf5c70879eda434884315cb2a06c9b3djoin_yr'
[2022-09-11 01:51:55,842] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaf5c70879eda434884315cb2a06c9b3djoin_yr%27
[2022-09-11 01:51:55,851] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.055077ms
[2022-09-11 01:51:56,882] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaf5c70879eda434884315cb2a06c9b3djoin_yr'
[2022-09-11 01:51:56,907] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaf5c70879eda434884315cb2a06c9b3djoin_yr%27
[2022-09-11 01:51:56,914] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.2323ms
[2022-09-11 01:51:57,945] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:51:57,970] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 01:51:57,974] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.702186ms
[2022-09-11 01:51:57,975] - [base_gsi:504] INFO - {'requestID': '74316966-3bea-481d-aa5d-3edf1f18787e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeaf5c70879eda434884315cb2a06c9b3djoin_yr', 'index_id': 'ce6353eab77ef4a1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.702186ms', 'executionTime': '2.637569ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 01:51:57,975] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:51:57,976] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:51:57,976] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:51:57,976] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:51:57,976] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:51:57,977] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:51:57,977] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:51:57,977] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:51:58,039] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:51:58,076] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:51:58,101] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 01:51:58,327] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 212.300144ms
[2022-09-11 01:51:58,327] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:51:58,328] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:51:59,560] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-11 01:51:59,610] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-11 01:51:59,639] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:51:59,666] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 01:51:59,669] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.182063ms
[2022-09-11 01:51:59,669] - [base_gsi:504] INFO - {'requestID': '9537deb0-b59d-4cec-b6a6-5c734d22ead3', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeaf5c70879eda434884315cb2a06c9b3djoin_yr', 'index_id': 'ce6353eab77ef4a1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.182063ms', 'executionTime': '2.110752ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 01:51:59,670] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:51:59,670] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:51:59,670] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:51:59,670] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:51:59,671] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:51:59,671] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:51:59,671] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:51:59,671] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:51:59,726] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:51:59,764] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:51:59,789] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 01:51:59,895] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.284194ms
[2022-09-11 01:51:59,895] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:51:59,896] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:52:01,110] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeaf5c70879eda434884315cb2a06c9b3djoin_yr ON default USING GSI
[2022-09-11 01:52:01,136] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeaf5c70879eda434884315cb2a06c9b3djoin_yr+ON+default+USING+GSI
[2022-09-11 01:52:01,189] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.62866ms
[2022-09-11 01:52:01,223] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaf5c70879eda434884315cb2a06c9b3djoin_yr'
[2022-09-11 01:52:01,249] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaf5c70879eda434884315cb2a06c9b3djoin_yr%27
[2022-09-11 01:52:01,256] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.336636ms
[2022-09-11 01:52:01,256] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b727e377-0b6c-4355-90bc-ffdf060c9693', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.336636ms', 'executionTime': '5.264527ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:52:01,367] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:52:01,371] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:01,371] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:02,067] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:02,124] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:52:02,124] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:52:02,240] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:52:02,301] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:52:02,304] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:02,304] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:03,013] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:03,068] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:52:03,068] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:52:03,182] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:52:03,237] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:52:03,238] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:52:03,238] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:52:03,264] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:52:03,292] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:52:03,301] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.587766ms
[2022-09-11 01:52:03,328] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:52:03,354] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:52:03,402] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.525167ms
[2022-09-11 01:52:03,474] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:52:03,474] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.43831269026315, 'mem_free': 12791230464, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:52:03,474] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:52:03,478] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:03,478] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:04,196] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:04,201] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:04,201] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:04,919] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:04,923] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:04,923] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:06,007] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:06,015] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:06,015] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:07,245] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:14,741] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-11 01:52:14,885] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:52:15,978] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:52:16,010] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:52:16,011] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:52:16,078] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:52:16,130] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:52:16,182] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:52:16,184] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:52:16,265] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:52:16,266] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:52:16,291] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:52:16,434] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:52:16,434] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:52:16,471] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:52:16,511] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:52:16,511] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:52:16,551] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:52:16,589] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:52:16,590] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:52:16,621] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:52:16,646] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:52:16,646] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:52:16,673] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:52:16,673] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 01:52:16,673] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:52:16,674] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 89.601s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_11'}
[2022-09-11 01:52:16,755] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:16,755] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:17,425] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:17,455] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:52:17,535] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:52:17,535] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-11 01:52:17,536] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:52:17,784] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:52:17,811] - [task:165] INFO -  {'uptime': '928', 'memoryTotal': 15466930176, 'memoryFree': 12956663808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:52:17,837] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:52:17,837] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:52:17,838] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:52:17,871] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:52:17,904] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:52:17,905] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:52:17,934] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:52:17,935] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:52:17,935] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:52:17,936] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:52:17,986] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:52:17,989] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:17,989] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:18,643] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:18,645] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:52:18,768] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:52:18,769] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:52:18,799] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:52:18,799] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:52:18,924] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:52:18,950] - [task:165] INFO -  {'uptime': '929', 'memoryTotal': 15466930176, 'memoryFree': 12959088640, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:52:18,975] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:52:19,004] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:52:19,004] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:52:19,054] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:52:19,058] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:19,059] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:19,703] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:19,704] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:52:19,830] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:52:19,831] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:52:19,861] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:52:19,862] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:52:19,984] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:52:20,012] - [task:165] INFO -  {'uptime': '924', 'memoryTotal': 15466930176, 'memoryFree': 12959121408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:52:20,038] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:52:20,066] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:52:20,066] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:52:20,116] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:52:20,119] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:20,119] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:20,823] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:20,824] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:52:20,956] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:52:20,957] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:52:20,991] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:52:20,992] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:52:21,112] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:52:21,140] - [task:165] INFO -  {'uptime': '923', 'memoryTotal': 15466930176, 'memoryFree': 12959457280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:52:21,167] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:52:21,197] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:52:21,197] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:52:21,250] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:52:21,253] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:21,254] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:21,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:21,911] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:52:22,040] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:52:22,041] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:52:22,071] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:52:22,071] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:52:22,158] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:52:22,719] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:52:27,724] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:52:27,816] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:52:27,820] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:27,820] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:28,532] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:28,533] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:52:28,667] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:52:28,668] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:52:28,668] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:52:29,318] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:52:29,373] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 01:52:29,374] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:52:30,265] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:52:30,576] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:52:30,858] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:52:30,860] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 01:52:30,916] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:30,917] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:31,601] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:31,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:31,605] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:32,610] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:32,620] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:32,621] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:33,758] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:33,768] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:33,768] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:34,931] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:41,780] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:52:41,781] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.05332166040272, 'mem_free': 12958752768, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:52:41,781] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:52:41,781] - [basetestcase:467] INFO - Time to execute basesetup : 25.02800750732422
[2022-09-11 01:52:41,832] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:52:41,832] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:52:41,885] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:52:41,885] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:52:41,938] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:52:41,939] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:52:41,991] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:52:41,992] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:52:42,042] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:52:42,106] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:52:42,106] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:52:42,107] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:52:47,119] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:52:47,123] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:52:47,124] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:52:47,798] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:52:48,975] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:52:49,140] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:52:52,258] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:52:52,338] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:52:52,339] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:52:52,339] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:53:22,366] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:53:22,396] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:53:22,422] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:53:22,513] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 89.207923ms
[2022-09-11 01:53:22,513] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '90a0831b-9f69-4e7d-ad56-de22d5b0a7d4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '89.207923ms', 'executionTime': '89.134379ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:53:22,514] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:53:22,551] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:53:22,577] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:53:23,213] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 634.383358ms
[2022-09-11 01:53:23,214] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:53:23,282] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:53:23,321] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:53:23,329] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.341809ms
[2022-09-11 01:53:23,546] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:53:23,580] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:53:23,597] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:53:23,599] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:53:23,613] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:53:23,679] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:53:24,503] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 01:53:24,528] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 01:53:24,569] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.098099ms
[2022-09-11 01:53:24,570] - [base_gsi:282] INFO - BUILD INDEX on default(`employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr`) USING GSI
[2022-09-11 01:53:25,600] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr`) USING GSI
[2022-09-11 01:53:25,626] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr%60%29+USING+GSI
[2022-09-11 01:53:25,648] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.66904ms
[2022-09-11 01:53:25,699] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr'
[2022-09-11 01:53:25,728] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr%27
[2022-09-11 01:53:25,738] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.752764ms
[2022-09-11 01:53:26,768] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr'
[2022-09-11 01:53:26,794] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr%27
[2022-09-11 01:53:26,801] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.828299ms
[2022-09-11 01:53:27,832] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr'
[2022-09-11 01:53:27,857] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr%27
[2022-09-11 01:53:27,865] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.281646ms
[2022-09-11 01:53:28,896] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:53:28,921] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 01:53:28,925] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.032864ms
[2022-09-11 01:53:28,926] - [base_gsi:504] INFO - {'requestID': 'e31c60f5-a9ce-460e-b5ff-fd83695f8151', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr', 'index_id': '13cfbc67a384b1ec', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.032864ms', 'executionTime': '2.974168ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 01:53:28,926] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 01:53:28,927] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:53:28,927] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 01:53:28,927] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 01:53:28,927] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:53:28,928] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:53:28,928] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:53:28,928] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 01:53:28,987] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 01:53:29,025] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 01:53:29,050] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 01:53:29,210] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 146.315085ms
[2022-09-11 01:53:29,210] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 01:53:29,211] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 01:53:30,860] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 01:53:34,005] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:53:34,060] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 01:53:34,061] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:53:34,995] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:53:35,304] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:53:35,565] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:53:35,567] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 01:53:37,600] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:53:37,600] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:37,628] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:53:37,656] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:53:37,656] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:37,683] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:53:37,711] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:53:37,711] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:37,737] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:53:37,765] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:53:37,765] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:37,792] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:53:37,842] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:53:37,867] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr'
[2022-09-11 01:53:37,892] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr%27
[2022-09-11 01:53:37,937] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.209107ms
[2022-09-11 01:53:37,937] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd66fa477-7db7-4fd7-8f7a-fcefd97e476d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '43.209107ms', 'executionTime': '43.134081ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:53:37,963] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr'
[2022-09-11 01:53:37,989] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d7e7f599044c958e4a18edf6a1b75ajoin_yr%27
[2022-09-11 01:53:37,992] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.262551ms
[2022-09-11 01:53:37,992] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8c4f3c97-b178-44a1-9848-9979b3b3696b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.262551ms', 'executionTime': '1.199608ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:53:38,042] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:53:38,170] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:53:38,172] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:38,172] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:38,875] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:38,934] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:53:38,934] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:53:39,044] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:53:39,100] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:53:39,103] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:39,103] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:39,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:39,875] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:53:39,875] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:53:39,999] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:53:40,059] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:53:40,059] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:53:40,059] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:53:40,085] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:53:40,111] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:53:40,120] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.295078ms
[2022-09-11 01:53:40,121] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '60aad442-9525-4d1f-bd7f-3b4710942e4d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.295078ms', 'executionTime': '8.217675ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:53:40,121] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:53:40,147] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:53:40,172] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:53:40,175] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.140017ms
[2022-09-11 01:53:40,175] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '154170cc-9203-4030-8219-d5597b21e7b0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.140017ms', 'executionTime': '1.087341ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:53:40,226] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:53:40,226] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.97556085889259, 'mem_free': 12941496320, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:53:40,226] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:53:40,229] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:40,230] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:40,947] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:40,952] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:40,952] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:41,729] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:41,740] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:41,740] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:42,950] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:42,958] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:42,958] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:44,245] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:51,838] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 01:53:51,979] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:53:52,292] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:53:52,320] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:53:52,320] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:53:52,381] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:53:52,435] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:53:52,490] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:53:52,490] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:53:52,574] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:53:52,575] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:53:52,600] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:52,739] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:53:52,739] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:52,771] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:53:52,797] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:53:52,797] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:52,824] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:53:52,849] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:53:52,849] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:52,875] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:53:52,900] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:53:52,900] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:52,926] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:53:52,926] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 01:53:52,926] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:53:52,927] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.225s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_01-36-44/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-11 01:53:52,985] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:52,985] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:53,704] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:53,738] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:53:53,817] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 01:53:53,817] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-11 01:53:53,818] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 01:53:54,013] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 01:53:54,041] - [task:165] INFO -  {'uptime': '1025', 'memoryTotal': 15466930176, 'memoryFree': 12940926976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:53:54,068] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 01:53:54,068] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 01:53:54,069] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 01:53:54,115] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 01:53:54,155] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 01:53:54,156] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 01:53:54,184] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 01:53:54,185] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 01:53:54,185] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 01:53:54,185] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 01:53:54,234] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:53:54,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:54,237] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:54,964] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:54,965] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:53:55,096] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:53:55,097] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:53:55,126] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:53:55,127] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:53:55,246] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 01:53:55,273] - [task:165] INFO -  {'uptime': '1024', 'memoryTotal': 15466930176, 'memoryFree': 12939235328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:53:55,298] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:53:55,328] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 01:53:55,328] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 01:53:55,379] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:53:55,381] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:55,382] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:56,109] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:56,110] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:53:56,256] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:53:56,257] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:53:56,288] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:53:56,289] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:53:56,410] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 01:53:56,438] - [task:165] INFO -  {'uptime': '1025', 'memoryTotal': 15466930176, 'memoryFree': 12939591680, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:53:56,465] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:53:56,492] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 01:53:56,492] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 01:53:56,542] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:53:56,545] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:56,545] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:57,238] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:57,239] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:53:57,383] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:53:57,384] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:53:57,414] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:53:57,414] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:53:57,529] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 01:53:57,555] - [task:165] INFO -  {'uptime': '1024', 'memoryTotal': 15466930176, 'memoryFree': 12939546624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 01:53:57,580] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 01:53:57,608] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 01:53:57,608] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 01:53:57,659] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 01:53:57,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:53:57,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:53:58,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:53:58,360] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:53:58,494] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:53:58,495] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:53:58,525] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 01:53:58,526] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 01:53:58,613] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 01:53:59,263] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:54:04,265] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 01:54:04,347] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 01:54:04,351] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:54:04,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:54:05,079] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:54:05,080] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 01:54:05,210] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 01:54:05,212] - [remote_util:5249] INFO - b'ok'
[2022-09-11 01:54:05,212] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 01:54:05,759] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 01:54:05,815] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 01:54:05,816] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 01:54:06,709] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:54:07,039] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:54:07,319] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 01:54:07,322] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-11 01:54:07,378] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:54:07,378] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:54:08,133] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:54:08,138] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:54:08,138] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:54:09,335] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:54:09,343] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:54:09,344] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:54:10,613] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:54:10,622] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:54:10,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:54:11,916] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:54:19,248] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:54:19,248] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.07750007600596, 'mem_free': 12933914624, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:54:19,248] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:54:19,249] - [basetestcase:467] INFO - Time to execute basesetup : 26.26667022705078
[2022-09-11 01:54:19,299] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:54:19,300] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:54:19,354] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:54:19,354] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:54:19,409] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:54:19,409] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:54:19,463] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 01:54:19,463] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 01:54:19,514] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:54:19,571] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 01:54:19,574] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 01:54:19,574] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 01:54:24,587] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 01:54:24,591] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:54:24,591] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:54:25,339] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:54:26,551] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 01:54:26,726] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 01:54:29,659] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 01:54:29,737] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:54:29,737] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 01:54:29,738] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 01:54:59,768] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 01:54:59,798] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:54:59,824] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:54:59,894] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.715759ms
[2022-09-11 01:54:59,894] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1c5bebb9-b342-4a7f-9307-93ce8af651dc', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.715759ms', 'executionTime': '67.641216ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 01:54:59,894] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 01:54:59,921] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 01:54:59,946] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 01:55:00,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 655.047173ms
[2022-09-11 01:55:00,604] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 01:55:00,658] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:55:00,703] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:55:00,711] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.439772ms
[2022-09-11 01:55:00,915] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:55:00,954] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 01:55:00,975] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 01:55:00,977] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 01:55:00,991] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 01:55:01,066] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:55:01,196] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:55:01,199] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:55:01,199] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:55:02,033] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:55:02,094] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:55:02,094] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 01:55:02,219] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:55:02,274] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:55:02,277] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:55:02,277] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:55:03,050] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:55:03,108] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 01:55:03,109] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 01:55:03,235] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 01:55:03,289] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 01:55:03,289] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 01:55:03,290] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:55:03,315] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 01:55:03,341] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 01:55:03,351] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.829259ms
[2022-09-11 01:55:03,376] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 01:55:03,401] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 01:55:03,451] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.763268ms
[2022-09-11 01:55:03,518] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 01:55:03,518] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.040504806454223, 'mem_free': 12762374144, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 01:55:03,518] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 01:55:03,522] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:55:03,522] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:55:04,264] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:55:04,269] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:55:04,269] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:55:05,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:55:05,496] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:55:05,496] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:55:06,879] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:55:06,888] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 01:55:06,888] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 01:55:08,209] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 01:55:15,908] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-11 01:55:16,053] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 01:55:17,272] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 01:55:17,300] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 01:55:17,300] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 01:55:17,361] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:55:17,414] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:55:17,467] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 01:55:17,468] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 01:55:17,550] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 01:55:17,551] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 01:55:17,576] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:55:17,705] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 01:55:17,705] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:55:17,735] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 01:55:17,762] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 01:55:17,762] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:55:17,788] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 01:55:17,813] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 01:55:17,813] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:55:17,839] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 01:55:17,865] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 01:55:17,866] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 01:55:17,892] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 01:55:17,892] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-11 01:55:17,892] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 01:55:17,893] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 84.989s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 27060 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-11.09.2022-00.06.cfg
Build Log: make-11.09.2022-00.06.log
Server Log: logs-11.09.2022-00.06.tar.gz

Finished