Building

Started building at 2022/09/10 22:04:43
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 10.09.2022-22.04
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at de1130b, changes since last good build: none
Switching plasma to unstable
plasma is at a338d0e, changes since last good build: 
fatal: Invalid revision range 70ba222be48a88c87394d7fa22445b17df4a5ec1..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable 60091cbe] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 674106f2] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 50e4bcd2] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable 840da433] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable cd2a493c] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 777bbbfe] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable 68602d65] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable 102b0004] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable 8defa36e] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 0c4aba15] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable 8818ff19] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 5b8f8668] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179227 at refs/changes/27/179227/10
Total 8 (delta 3), reused 4 (delta 3)
[unstable ac485be7] MB-51825: Remove vbuckets arg from indexer command line and its usage
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Aug 25 18:17:32 2022 +0530
 2 files changed, 9 deletions(-)
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 1a54b1be] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable d4663e8] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable d620f52] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 3207924] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable f4fec46] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 9772d3e] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable f7b5c60] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable 41e323f] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 67d5c96] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/10 22:58:59
Testing mode: sanity,unit,functional,integration
Using storage type: plasma
Setting ulimit to 200000

Simple Test

Sep 10 23:04:24 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 23:08:16 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 23:09:01 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 10 23:10:32 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 10 23:11:26 b" 'stop_on_failure': 'True'}"
Sep 10 23:11:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 23:11:26 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_1'}"
Sep 10 23:11:26 b'-->result: '
Sep 10 23:11:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 10 23:11:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 23:11:26 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_2'}"
Sep 10 23:11:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 23:11:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 10 23:11:26 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_3'}"
Sep 10 23:11:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 23:11:26 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 23:11:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 10 23:11:26 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_4'}"
Sep 10 23:11:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 23:11:26 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 23:11:26 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 23:21:12 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 10 23:21:58 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 10 23:25:44 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 23:29:42 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 23:32:26 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 23:32:26 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_5'}"
Sep 10 23:32:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 23:32:26 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 10 23:32:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 10 23:32:26 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_6'}"
Sep 10 23:32:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 23:32:26 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 23:32:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 10 23:32:26 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_7'}"
Sep 10 23:32:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 23:32:26 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 23:32:26 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 10 23:32:26 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 10 23:32:26 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_22-59-21/test_8'}"
Sep 10 23:32:26 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 23:32:26 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 23:32:26 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 23:32:26 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 10 23:32:26 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 10 23:32:27 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 23:32:27 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 23:32:27 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 10 23:32:27 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 10 23:32:27 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 10 23:32:27 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 10 23:32:27 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 10 23:32:27 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 187: go: command not found
/home/buildbot/bin/dotest: line 190: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=plasma
Printing gsi_type=plasma
gsi_type=plasma
In here
-p makefile=True,gsi_type=plasma
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'plasma',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-10 23:35:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:19 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 23:35:22 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 23:35:28 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:43 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 23:35:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:48 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-10 23:35:49 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 2113511424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:35:49 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 23:35:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:35:50 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 1945890816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:35:50 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 23:35:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:35:51 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1813180416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:35:51 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 23:35:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:35:52 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1714024448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:35:52 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:35:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:35:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:35:53 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 23:35:53 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 23:35:53 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-10 23:35:54 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:35:59 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 23:35:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 23:35:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-10 23:35:59 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 23:36:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 23:36:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 23:36:01 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 23:36:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 49.62071388037771, 'mem_free': 13410570240, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 23:36:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 23:36:04 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 45.14138913154602
2022-09-10 23:36:04 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:36:04 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 23:36:05 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 23:36:10 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 23:36:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:11 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 23:36:11 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 23:36:13 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 23:36:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:13 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 23:36:13 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 23:36:43 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 23:36:43 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 23:36:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 23:36:43 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 125.904651ms
2022-09-10 23:36:43 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '7b38f3d2-5c1d-4f57-9c8a-07b2c9388f3f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '125.904651ms', 'executionTime': '125.682412ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 23:36:43 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 23:36:43 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 23:36:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 738.959321ms
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.633538ms
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.624192ms
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 48.174667ms
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.501575328165035, 'mem_free': 13374611456, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:50 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 23:36:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 23:36:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 23:36:51 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 92.133s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-10 23:36:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.5695116290207, 'mem_free': 13345660928, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 23:36:57 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 23:36:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-10 23:36:58 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 23:36:58 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13345660928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 23:36:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13341696000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:36:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13333729280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13341134848, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 23:37:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 23:37:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-10 23:37:01 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 23:37:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 23:37:07 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 23:37:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 23:37:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-10 23:37:08 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 23:37:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 23:37:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 23:37:10 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.89800475233734, 'mem_free': 13331816448, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 22.430574655532837
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 23:37:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 23:37:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 23:37:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:37:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:37:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:37:20 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 23:37:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 23:37:22 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 23:37:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:37:22 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 23:37:22 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 23:37:52 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 23:37:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 23:37:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 23:37:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.767919ms
2022-09-10 23:37:52 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '98ed7753-41e9-4a98-8fcb-c4ac24f04d3d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.767919ms', 'executionTime': '68.705733ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 23:37:52 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 23:37:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 23:37:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 23:37:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 733.533941ms
2022-09-10 23:37:53 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 23:37:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 23:37:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 23:37:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.071329ms
2022-09-10 23:37:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:37:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 23:37:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 23:37:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 23:37:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 23:37:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:37:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee89f02e2f2c5c4e838dee89a1ad0e913djob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-10 23:37:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee89f02e2f2c5c4e838dee89a1ad0e913djob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-10 23:37:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 46.082243ms
2022-09-10 23:37:55 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee89f02e2f2c5c4e838dee89a1ad0e913djob_title) USING GSI
2022-09-10 23:37:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee89f02e2f2c5c4e838dee89a1ad0e913djob_title) USING GSI
2022-09-10 23:37:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee89f02e2f2c5c4e838dee89a1ad0e913djob_title%29+USING+GSI
2022-09-10 23:37:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 19.745721ms
2022-09-10 23:37:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee89f02e2f2c5c4e838dee89a1ad0e913djob_title'
2022-09-10 23:37:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee89f02e2f2c5c4e838dee89a1ad0e913djob_title%27
2022-09-10 23:37:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.431538ms
2022-09-10 23:37:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee89f02e2f2c5c4e838dee89a1ad0e913djob_title'
2022-09-10 23:37:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee89f02e2f2c5c4e838dee89a1ad0e913djob_title%27
2022-09-10 23:37:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.945387ms
2022-09-10 23:37:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 23:37:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 23:37:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-10 23:37:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.587208ms
2022-09-10 23:37:59 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '26817c13-a863-4278-8131-0b1307ed95a9', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee89f02e2f2c5c4e838dee89a1ad0e913djob_title', 'index_id': '924ff01e84f83d8f', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.587208ms', 'executionTime': '2.518129ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-10 23:37:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 23:37:59 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 23:37:59 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 23:37:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-10 23:37:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-10 23:37:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-10 23:37:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-10 23:37:59 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-10 23:37:59 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-10 23:38:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 23:38:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 23:38:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-10 23:38:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 144.248525ms
2022-09-10 23:38:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-10 23:38:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-10 23:38:01 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 23:38:01 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee89f02e2f2c5c4e838dee89a1ad0e913djob_title'
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee89f02e2f2c5c4e838dee89a1ad0e913djob_title%27
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.531755ms
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee89f02e2f2c5c4e838dee89a1ad0e913djob_title ON default USING GSI
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee89f02e2f2c5c4e838dee89a1ad0e913djob_title+ON+default+USING+GSI
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 45.628555ms
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee89f02e2f2c5c4e838dee89a1ad0e913djob_title'
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee89f02e2f2c5c4e838dee89a1ad0e913djob_title%27
2022-09-10 23:38:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.888048ms
2022-09-10 23:38:02 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '343961ee-cbae-4357-acea-5b1baa3e9f53', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.888048ms', 'executionTime': '5.82484ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 23:38:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:38:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:38:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.229404ms
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 61.312259ms
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.69118150568498, 'mem_free': 13211951104, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:38:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 23:38:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 23:38:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 23:38:08 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 23:38:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 23:38:09 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 23:38:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.886s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_2'}
[2022-09-10 23:38:09,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:09,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:10,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:10,137] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:38:10,214] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:38:10,214] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-10 23:38:10,215] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:38:10,864] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:38:10,893] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13255245824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:38:10,920] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:38:10,920] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:38:10,921] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:38:10,950] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:38:10,994] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:38:10,995] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:38:11,024] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:38:11,025] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:38:11,025] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:38:11,025] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:38:11,076] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:38:11,079] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:11,079] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:11,363] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:11,364] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:38:11,439] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:38:11,440] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:38:11,469] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:11,497] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:11,526] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:38:11,648] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:38:11,677] - [task:165] INFO -  {'uptime': '164', 'memoryTotal': 15466930176, 'memoryFree': 13223723008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:38:11,702] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:38:11,729] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:38:11,730] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:38:11,780] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:38:11,783] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:11,783] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:12,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:12,057] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:38:12,125] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:38:12,126] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:38:12,156] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:12,183] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:12,214] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:38:12,330] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:38:12,357] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13223497728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:38:12,382] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:38:12,411] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:38:12,411] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:38:12,462] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:38:12,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:12,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:12,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:12,725] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:38:12,795] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:38:12,796] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:38:12,826] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:12,851] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:12,879] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:38:12,991] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:38:13,020] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13223690240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:38:13,046] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:38:13,073] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:38:13,074] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:38:13,123] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:38:13,126] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:13,126] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:13,392] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:13,394] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:38:13,461] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:38:13,463] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:38:13,494] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:13,520] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:38:13,548] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:38:13,640] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:38:14,297] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:38:19,302] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:38:19,388] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:38:19,393] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:19,393] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:19,677] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:19,678] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:38:19,747] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:38:19,748] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:38:19,749] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:38:20,775] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:38:20,829] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 23:38:20,829] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:38:21,838] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:38:22,169] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:38:22,505] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:38:22,507] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:38:22,564] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:22,564] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:22,841] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:22,846] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:22,846] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:23,126] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:23,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:23,132] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:23,384] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:23,389] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:23,389] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:23,800] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:27,587] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:38:27,588] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.56069837210151, 'mem_free': 13248421888, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:38:27,588] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:38:27,588] - [basetestcase:467] INFO - Time to execute basesetup : 17.757342100143433
[2022-09-10 23:38:27,644] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:38:27,645] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:38:27,703] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:38:27,703] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:38:27,761] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:38:27,762] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:38:27,815] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:38:27,816] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:38:27,870] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:38:27,929] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:38:27,930] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:38:27,930] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:38:32,939] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:38:32,942] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:38:32,943] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:38:33,197] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:38:34,146] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:38:34,312] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:38:36,927] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:38:36,949] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 23:38:37,609] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:38:37,609] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:38:37,610] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:39:07,637] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:39:07,667] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:39:07,693] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:39:07,761] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.384175ms
[2022-09-10 23:39:07,761] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5fd38214-1e11-4bf7-b448-69ada87216f9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.384175ms', 'executionTime': '66.326447ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:39:07,761] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:39:07,788] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:39:07,813] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:39:08,551] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 735.85764ms
[2022-09-10 23:39:08,551] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:39:08,611] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:39:08,655] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:39:08,663] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.340231ms
[2022-09-10 23:39:08,857] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:39:08,900] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:39:08,931] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:39:08,931] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:39:08,956] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:39:09,014] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:39:09,819] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb3a2d994faa24f568547f96fe0e7fb56job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:39:09,845] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb3a2d994faa24f568547f96fe0e7fb56job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:39:09,907] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.761916ms
[2022-09-10 23:39:09,908] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb3a2d994faa24f568547f96fe0e7fb56job_title) USING GSI
[2022-09-10 23:39:10,941] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb3a2d994faa24f568547f96fe0e7fb56job_title) USING GSI
[2022-09-10 23:39:10,969] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb3a2d994faa24f568547f96fe0e7fb56job_title%29+USING+GSI
[2022-09-10 23:39:10,994] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.800536ms
[2022-09-10 23:39:12,025] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb3a2d994faa24f568547f96fe0e7fb56job_title'
[2022-09-10 23:39:12,051] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb3a2d994faa24f568547f96fe0e7fb56job_title%27
[2022-09-10 23:39:12,061] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.111699ms
[2022-09-10 23:39:13,090] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb3a2d994faa24f568547f96fe0e7fb56job_title'
[2022-09-10 23:39:13,116] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb3a2d994faa24f568547f96fe0e7fb56job_title%27
[2022-09-10 23:39:13,125] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.051277ms
[2022-09-10 23:39:13,571] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 23:39:13,744] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:39:14,633] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:39:15,128] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:39:15,162] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:39:15,189] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 23:39:15,192] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.06415ms
[2022-09-10 23:39:15,193] - [task:3246] INFO - {'requestID': 'b3247fb0-998e-475c-bd3a-2dcba87f5dab', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb3a2d994faa24f568547f96fe0e7fb56job_title', 'index_id': 'c0d9576bc2868607', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.06415ms', 'executionTime': '1.990922ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 23:39:15,193] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:39:15,193] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:39:15,194] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:39:15,194] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:39:15,194] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 23:39:15,194] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:39:15,195] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:39:15,195] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:39:15,195] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:39:16,194] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:39:16,225] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:39:16,251] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 23:39:16,356] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 94.865853ms
[2022-09-10 23:39:16,356] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:39:16,357] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:39:17,118] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:39:17,119] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:39:18,150] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb3a2d994faa24f568547f96fe0e7fb56job_title'
[2022-09-10 23:39:18,177] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb3a2d994faa24f568547f96fe0e7fb56job_title%27
[2022-09-10 23:39:18,186] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.591342ms
[2022-09-10 23:39:18,214] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb3a2d994faa24f568547f96fe0e7fb56job_title ON default USING GSI
[2022-09-10 23:39:18,242] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeb3a2d994faa24f568547f96fe0e7fb56job_title+ON+default+USING+GSI
[2022-09-10 23:39:18,283] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.844813ms
[2022-09-10 23:39:18,322] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb3a2d994faa24f568547f96fe0e7fb56job_title'
[2022-09-10 23:39:18,350] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb3a2d994faa24f568547f96fe0e7fb56job_title%27
[2022-09-10 23:39:18,361] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.080581ms
[2022-09-10 23:39:18,361] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3c581f46-a15b-4442-bba6-a13573b7d9b6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.080581ms', 'executionTime': '8.008867ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:39:18,469] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:39:18,472] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:18,473] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:18,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:18,908] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:39:18,909] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:39:18,975] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:39:19,032] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:39:19,035] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:19,036] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:19,422] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:19,483] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:39:19,483] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:39:19,548] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:39:19,605] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:39:19,605] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:39:19,606] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:39:19,632] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:39:19,660] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:39:19,671] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.194379ms
[2022-09-10 23:39:19,700] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:39:19,727] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:39:19,782] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.889799ms
[2022-09-10 23:39:19,861] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:39:19,861] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.42706660501646, 'mem_free': 13116452864, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:39:19,861] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:39:19,866] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:19,866] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:20,235] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:20,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:20,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:20,587] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:20,592] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:20,592] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:21,222] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:21,229] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:21,229] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:21,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:25,747] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:39:25,890] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:39:26,400] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:39:26,427] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:39:26,428] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:39:26,483] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:39:26,535] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:39:26,587] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:39:26,588] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:39:26,666] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:39:26,667] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:39:26,692] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:39:26,819] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:39:26,819] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:39:26,845] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:39:26,872] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:39:26,872] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:39:26,898] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:39:26,923] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:39:26,924] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:39:26,951] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:39:26,976] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:39:26,977] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:39:27,008] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:39:27,009] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:39:27,009] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:39:27,010] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 77.235s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_3'}
[2022-09-10 23:39:27,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:27,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:27,433] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:27,465] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:39:27,546] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:39:27,546] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-10 23:39:27,547] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:39:28,142] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:39:28,171] - [task:165] INFO -  {'uptime': '244', 'memoryTotal': 15466930176, 'memoryFree': 13116452864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:39:28,197] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:39:28,197] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:39:28,198] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:39:28,239] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:39:28,276] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:39:28,277] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:39:28,307] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:39:28,308] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:39:28,308] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:39:28,308] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:39:28,358] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:39:28,361] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:28,361] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:28,681] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:28,682] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:39:28,762] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:39:28,763] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:39:28,794] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:28,822] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:28,850] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:39:28,966] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:39:28,994] - [task:165] INFO -  {'uptime': '244', 'memoryTotal': 15466930176, 'memoryFree': 13192876032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:39:29,021] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:39:29,050] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:39:29,050] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:39:29,102] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:39:29,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:29,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:29,423] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:29,424] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:39:29,499] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:39:29,500] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:39:29,531] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:29,558] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:29,586] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:39:29,702] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:39:29,729] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13193740288, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:39:29,755] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:39:29,782] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:39:29,783] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:39:29,834] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:39:29,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:29,839] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:30,158] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:30,159] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:39:30,235] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:39:30,236] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:39:30,267] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:30,295] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:30,323] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:39:30,442] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:39:30,472] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13192683520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:39:30,498] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:39:30,526] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:39:30,527] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:39:30,578] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:39:30,581] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:30,581] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:30,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:30,911] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:39:30,992] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:39:30,993] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:39:31,026] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:31,054] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:39:31,084] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:39:31,173] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:39:31,769] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:39:36,772] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:39:36,857] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:39:36,861] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:36,862] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:37,181] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:37,182] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:39:37,257] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:39:37,258] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:39:37,259] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:39:38,285] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:39:38,343] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 23:39:38,344] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:39:39,242] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:39:39,548] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:39:39,801] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:39:39,803] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:39:39,857] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:39,857] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:40,165] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:40,170] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:40,170] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:40,472] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:40,477] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:40,477] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:40,777] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:40,783] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:40,783] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:41,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:45,359] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:39:45,360] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.24550896646938, 'mem_free': 13193441280, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:39:45,360] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:39:45,360] - [basetestcase:467] INFO - Time to execute basesetup : 18.2504403591156
[2022-09-10 23:39:45,410] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:39:45,410] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:39:45,464] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:39:45,464] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:39:45,521] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:39:45,521] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:39:45,575] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:39:45,575] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:39:45,626] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:39:45,697] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:39:45,698] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:39:45,698] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:39:50,710] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:39:50,713] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:39:50,714] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:39:51,028] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:39:51,981] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:39:52,164] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:39:54,755] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:39:54,776] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 23:39:55,597] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:39:55,597] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:39:55,597] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:40:25,614] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:40:25,643] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:40:25,669] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:40:25,738] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.043651ms
[2022-09-10 23:40:25,738] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b169bf09-1216-45e6-b858-3f9a40926ee3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.043651ms', 'executionTime': '66.982951ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:40:25,738] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:40:25,764] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:40:25,789] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:40:26,476] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 684.894205ms
[2022-09-10 23:40:26,476] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:40:26,546] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:40:26,576] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:40:26,584] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.950728ms
[2022-09-10 23:40:26,790] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:40:26,827] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:40:26,839] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:40:26,840] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:40:26,861] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:40:26,933] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:40:27,745] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4d35750ba7c14271a90a0c2f210a1cc9job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:40:27,770] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4d35750ba7c14271a90a0c2f210a1cc9job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:40:27,825] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.438422ms
[2022-09-10 23:40:27,826] - [base_gsi:282] INFO - BUILD INDEX on default(employee4d35750ba7c14271a90a0c2f210a1cc9job_title) USING GSI
[2022-09-10 23:40:28,857] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee4d35750ba7c14271a90a0c2f210a1cc9job_title) USING GSI
[2022-09-10 23:40:28,883] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee4d35750ba7c14271a90a0c2f210a1cc9job_title%29+USING+GSI
[2022-09-10 23:40:28,910] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.247359ms
[2022-09-10 23:40:29,940] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d35750ba7c14271a90a0c2f210a1cc9job_title'
[2022-09-10 23:40:29,966] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d35750ba7c14271a90a0c2f210a1cc9job_title%27
[2022-09-10 23:40:29,975] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.1062ms
[2022-09-10 23:40:31,007] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d35750ba7c14271a90a0c2f210a1cc9job_title'
[2022-09-10 23:40:31,032] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d35750ba7c14271a90a0c2f210a1cc9job_title%27
[2022-09-10 23:40:31,039] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.3795ms
[2022-09-10 23:40:31,415] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 23:40:31,673] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:40:32,425] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:40:33,041] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:40:33,075] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:40:33,101] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 23:40:33,105] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.389952ms
[2022-09-10 23:40:33,105] - [task:3246] INFO - {'requestID': '17dc9c33-1057-4a4d-ae0c-84f7a9683fec', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4d35750ba7c14271a90a0c2f210a1cc9job_title', 'index_id': '4fe081e6cb0d0946', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.389952ms', 'executionTime': '2.31467ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 23:40:33,106] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:40:33,106] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:40:33,106] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:40:33,107] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:40:33,107] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 23:40:33,107] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:40:33,107] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:40:33,108] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:40:33,108] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:40:34,106] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:40:34,136] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:40:34,161] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 23:40:34,282] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 111.717133ms
[2022-09-10 23:40:34,282] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:40:34,283] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:40:35,045] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:40:35,045] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:40:36,076] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d35750ba7c14271a90a0c2f210a1cc9job_title'
[2022-09-10 23:40:36,103] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d35750ba7c14271a90a0c2f210a1cc9job_title%27
[2022-09-10 23:40:36,113] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.511892ms
[2022-09-10 23:40:36,139] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4d35750ba7c14271a90a0c2f210a1cc9job_title ON default USING GSI
[2022-09-10 23:40:36,165] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4d35750ba7c14271a90a0c2f210a1cc9job_title+ON+default+USING+GSI
[2022-09-10 23:40:36,216] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.145635ms
[2022-09-10 23:40:36,249] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d35750ba7c14271a90a0c2f210a1cc9job_title'
[2022-09-10 23:40:36,276] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d35750ba7c14271a90a0c2f210a1cc9job_title%27
[2022-09-10 23:40:36,285] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.170499ms
[2022-09-10 23:40:36,285] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '63aeaeef-a847-423d-9c36-b27e93e96536', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.170499ms', 'executionTime': '7.083823ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:40:36,393] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:40:36,396] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:36,396] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:36,780] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:36,835] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:40:36,836] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:40:36,907] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:40:36,964] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:40:36,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:36,967] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:37,389] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:37,451] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:40:37,451] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:40:37,521] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:40:37,583] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:40:37,583] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:40:37,583] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:40:37,610] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:40:37,637] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:40:37,648] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.829419ms
[2022-09-10 23:40:37,676] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:40:37,705] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:40:37,752] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.334304ms
[2022-09-10 23:40:37,830] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:40:37,830] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.33710807517165, 'mem_free': 13065818112, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:40:37,830] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:40:37,835] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:37,835] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:38,261] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:38,267] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:38,267] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:38,855] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:38,866] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:38,866] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:39,605] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:39,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:39,617] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:40,311] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:44,552] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:40:44,697] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:40:45,844] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:40:45,871] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:40:45,872] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:40:45,930] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:40:45,984] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:40:46,036] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:40:46,037] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:40:46,115] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:40:46,116] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:40:46,143] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:40:46,275] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:40:46,275] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:40:46,302] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:40:46,327] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:40:46,327] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:40:46,353] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:40:46,378] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:40:46,379] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:40:46,405] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:40:46,431] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:40:46,431] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:40:46,458] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:40:46,459] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:40:46,459] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:40:46,459] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.401s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_4'}
[2022-09-10 23:40:46,559] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:46,559] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:46,941] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:46,971] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:40:47,053] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:40:47,054] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-10 23:40:47,054] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:40:47,591] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:40:47,619] - [task:165] INFO -  {'uptime': '324', 'memoryTotal': 15466930176, 'memoryFree': 13076082688, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:40:47,644] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:40:47,645] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:40:47,645] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:40:47,683] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:40:47,719] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:40:47,719] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:40:47,748] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:40:47,749] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:40:47,749] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:40:47,749] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:40:47,798] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:40:47,800] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:47,800] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:48,150] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:48,151] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:40:48,232] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:40:48,234] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:40:48,264] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:48,290] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:48,319] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:40:48,438] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:40:48,465] - [task:165] INFO -  {'uptime': '320', 'memoryTotal': 15466930176, 'memoryFree': 13149188096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:40:48,490] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:40:48,518] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:40:48,518] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:40:48,569] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:40:48,572] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:48,572] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:48,931] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:48,932] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:40:49,016] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:40:49,018] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:40:49,047] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:49,072] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:49,099] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:40:49,216] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:40:49,243] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13148901376, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:40:49,270] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:40:49,297] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:40:49,297] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:40:49,347] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:40:49,351] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:49,351] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:49,709] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:49,710] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:40:49,798] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:40:49,799] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:40:49,829] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:49,855] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:49,885] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:40:50,002] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:40:50,029] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13147181056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:40:50,055] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:40:50,082] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:40:50,083] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:40:50,133] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:40:50,136] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:50,136] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:50,488] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:50,489] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:40:50,570] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:40:50,571] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:40:50,598] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:50,626] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:40:50,655] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:40:50,746] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:40:51,339] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:40:56,345] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:40:56,430] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:40:56,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:56,436] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:56,842] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:56,843] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:40:56,935] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:40:56,936] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:40:56,937] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:40:57,882] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:40:57,938] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 23:40:57,938] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:40:58,806] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:40:59,112] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:40:59,380] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:40:59,382] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:40:59,439] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:59,439] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:40:59,835] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:40:59,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:40:59,840] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:00,213] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:00,218] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:00,218] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:00,815] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:00,824] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:00,825] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:01,478] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:06,041] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:41:06,041] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.87370747710931, 'mem_free': 13153611776, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:41:06,041] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:41:06,042] - [basetestcase:467] INFO - Time to execute basesetup : 19.484941720962524
[2022-09-10 23:41:06,095] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:41:06,096] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:41:06,152] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:41:06,152] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:41:06,207] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:41:06,208] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:41:06,263] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:41:06,263] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:41:06,317] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:41:06,375] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:41:06,375] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:41:06,375] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:41:11,384] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:41:11,388] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:11,388] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:11,779] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:12,764] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:41:13,016] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:41:15,482] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:41:15,497] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 23:41:16,182] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:41:16,183] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:41:16,183] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:41:46,195] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:41:46,224] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:41:46,249] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:41:46,316] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.294604ms
[2022-09-10 23:41:46,316] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4a176a37-3b89-4c6e-a47c-9bbf817cbb11', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.294604ms', 'executionTime': '65.241929ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:41:46,317] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:41:46,342] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:41:46,367] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:41:47,121] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 751.957253ms
[2022-09-10 23:41:47,121] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:41:47,178] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:41:47,225] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:41:47,234] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.495843ms
[2022-09-10 23:41:47,420] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:41:47,454] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:41:47,467] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:41:47,467] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:41:47,485] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:41:47,563] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:41:48,384] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeebe659d4f175a4bf9a62a406d80781059job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:41:48,410] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeebe659d4f175a4bf9a62a406d80781059job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:41:48,503] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 91.241924ms
[2022-09-10 23:41:48,504] - [base_gsi:282] INFO - BUILD INDEX on default(employeebe659d4f175a4bf9a62a406d80781059job_title) USING GSI
[2022-09-10 23:41:49,534] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeebe659d4f175a4bf9a62a406d80781059job_title) USING GSI
[2022-09-10 23:41:49,561] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeebe659d4f175a4bf9a62a406d80781059job_title%29+USING+GSI
[2022-09-10 23:41:49,583] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.693301ms
[2022-09-10 23:41:50,613] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebe659d4f175a4bf9a62a406d80781059job_title'
[2022-09-10 23:41:50,638] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebe659d4f175a4bf9a62a406d80781059job_title%27
[2022-09-10 23:41:50,647] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.715371ms
[2022-09-10 23:41:51,678] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebe659d4f175a4bf9a62a406d80781059job_title'
[2022-09-10 23:41:51,704] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebe659d4f175a4bf9a62a406d80781059job_title%27
[2022-09-10 23:41:51,710] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.869741ms
[2022-09-10 23:41:52,180] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 23:41:52,350] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:41:53,220] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:41:53,409] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-10 23:41:53,409] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:41:53,593] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-10 23:41:53,593] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-10 23:41:53,593] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2228913353, 0, b'')
[2022-09-10 23:41:53,713] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:41:53,742] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:41:53,770] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 23:41:53,775] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.043808ms
[2022-09-10 23:41:53,776] - [task:3246] INFO - {'requestID': 'fb766045-ec64-4ef0-8c08-282511397611', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeebe659d4f175a4bf9a62a406d80781059job_title', 'index_id': 'dd5c850cb1269afc', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.043808ms', 'executionTime': '2.986728ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 23:41:53,776] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:41:53,777] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:41:53,780] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:41:53,780] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:41:53,781] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 23:41:53,781] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:41:53,781] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:41:53,782] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:41:53,782] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:41:54,781] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:41:54,811] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:41:54,837] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 23:41:54,973] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 128.315049ms
[2022-09-10 23:41:54,974] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:41:54,974] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:41:55,725] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:41:55,725] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:41:56,757] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebe659d4f175a4bf9a62a406d80781059job_title'
[2022-09-10 23:41:56,784] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebe659d4f175a4bf9a62a406d80781059job_title%27
[2022-09-10 23:41:56,792] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.615152ms
[2022-09-10 23:41:56,819] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeebe659d4f175a4bf9a62a406d80781059job_title ON default USING GSI
[2022-09-10 23:41:56,844] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeebe659d4f175a4bf9a62a406d80781059job_title+ON+default+USING+GSI
[2022-09-10 23:41:56,885] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.888101ms
[2022-09-10 23:41:56,930] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebe659d4f175a4bf9a62a406d80781059job_title'
[2022-09-10 23:41:56,972] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebe659d4f175a4bf9a62a406d80781059job_title%27
[2022-09-10 23:41:56,982] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.464994ms
[2022-09-10 23:41:56,982] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '56b1b53e-ac3d-4796-ac5c-0d0b2cdc5e68', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.464994ms', 'executionTime': '7.399945ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:41:57,092] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:41:57,095] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:57,095] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:57,578] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:57,636] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:41:57,636] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:41:57,715] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:41:57,774] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:41:57,777] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:57,777] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:58,252] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:58,314] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:41:58,314] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:41:58,391] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:41:58,450] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:41:58,450] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:41:58,450] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:41:58,476] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:41:58,503] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:41:58,512] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.891033ms
[2022-09-10 23:41:58,539] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:41:58,564] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:41:58,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.318252ms
[2022-09-10 23:41:58,690] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:41:58,690] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 9.917252361805067, 'mem_free': 13013606400, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:41:58,690] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:41:58,694] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:58,694] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:59,148] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:59,153] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:59,153] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:41:59,847] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:41:59,859] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:41:59,859] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:00,659] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:00,667] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:00,667] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:01,450] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:06,118] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:42:06,266] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:42:07,430] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:42:07,458] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:42:07,458] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:42:07,518] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:42:07,570] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:42:07,621] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:42:07,622] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:42:07,699] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:42:07,700] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:42:07,725] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:42:07,851] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:42:07,851] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:42:07,877] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:42:07,907] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:42:07,907] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:42:07,934] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:42:07,959] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:42:07,959] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:42:07,985] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:42:08,010] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:42:08,010] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:42:08,037] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:42:08,037] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:42:08,037] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:42:08,038] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 81.532s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_5'}
[2022-09-10 23:42:08,138] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:08,138] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:08,568] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:08,599] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:42:08,681] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:42:08,682] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-10 23:42:08,682] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:42:09,166] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:42:09,192] - [task:165] INFO -  {'uptime': '405', 'memoryTotal': 15466930176, 'memoryFree': 13108666368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:42:09,218] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:42:09,218] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:42:09,218] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:42:09,247] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:42:09,283] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:42:09,284] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:42:09,311] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:42:09,312] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:42:09,312] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:42:09,312] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:42:09,365] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:42:09,368] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:09,368] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:09,791] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:09,793] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:42:09,886] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:42:09,887] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:42:09,918] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:09,945] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:09,975] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:42:10,100] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:42:10,127] - [task:165] INFO -  {'uptime': '405', 'memoryTotal': 15466930176, 'memoryFree': 13108989952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:42:10,154] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:42:10,183] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:42:10,184] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:42:10,238] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:42:10,241] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:10,241] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:10,663] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:10,664] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:42:10,753] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:42:10,754] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:42:10,785] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:10,810] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:10,839] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:42:10,957] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:42:10,984] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13108436992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:42:11,009] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:42:11,038] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:42:11,038] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:42:11,088] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:42:11,093] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:11,093] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:11,509] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:11,510] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:42:11,597] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:42:11,598] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:42:11,628] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:11,655] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:11,686] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:42:11,799] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:42:11,825] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13108260864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:42:11,850] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:42:11,877] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:42:11,877] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:42:11,927] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:42:11,931] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:11,931] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:12,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:12,331] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:42:12,420] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:42:12,422] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:42:12,454] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:12,481] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:42:12,510] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:42:12,602] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:42:13,136] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:42:18,142] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:42:18,227] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:42:18,231] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:18,232] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:18,676] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:18,677] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:42:18,770] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:42:18,771] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:42:18,771] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:42:19,714] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:42:19,767] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 23:42:19,768] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:42:20,661] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:42:20,972] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:42:21,248] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:42:21,250] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:42:21,305] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:21,305] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:21,729] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:21,734] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:21,734] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:22,152] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:22,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:22,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:22,865] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:22,873] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:22,874] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:23,632] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:28,675] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:42:28,675] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.44224251405561, 'mem_free': 13113257984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:42:28,675] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:42:28,675] - [basetestcase:467] INFO - Time to execute basesetup : 20.53963875770569
[2022-09-10 23:42:28,726] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:42:28,726] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:42:28,780] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:42:28,780] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:42:28,834] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:42:28,834] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:42:28,887] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:42:28,887] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:42:28,938] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:42:29,003] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:42:29,003] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:42:29,004] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:42:34,011] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:42:34,015] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:42:34,015] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:42:34,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:42:35,404] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:42:35,652] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:42:38,113] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:42:38,130] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 23:42:39,189] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:42:39,190] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:42:39,190] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:43:09,211] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:43:09,240] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:43:09,265] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:43:09,334] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.60428ms
[2022-09-10 23:43:09,334] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '13ea09cd-1a43-45ad-9bee-47fd9c55d2b3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.60428ms', 'executionTime': '66.545611ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:43:09,334] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:43:09,362] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:43:09,389] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:43:10,133] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 741.303725ms
[2022-09-10 23:43:10,133] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:43:10,192] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:43:10,244] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:43:10,253] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.504236ms
[2022-09-10 23:43:10,449] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:43:10,484] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:43:10,500] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:43:10,500] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:43:10,518] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:43:10,605] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:43:11,411] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0157a6f4c941447eae72408497838ee2job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:43:11,437] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0157a6f4c941447eae72408497838ee2job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:43:11,504] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.230513ms
[2022-09-10 23:43:11,504] - [base_gsi:282] INFO - BUILD INDEX on default(employee0157a6f4c941447eae72408497838ee2job_title) USING GSI
[2022-09-10 23:43:12,535] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee0157a6f4c941447eae72408497838ee2job_title) USING GSI
[2022-09-10 23:43:12,560] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee0157a6f4c941447eae72408497838ee2job_title%29+USING+GSI
[2022-09-10 23:43:12,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.379594ms
[2022-09-10 23:43:13,617] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0157a6f4c941447eae72408497838ee2job_title'
[2022-09-10 23:43:13,642] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0157a6f4c941447eae72408497838ee2job_title%27
[2022-09-10 23:43:13,652] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.520583ms
[2022-09-10 23:43:14,684] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0157a6f4c941447eae72408497838ee2job_title'
[2022-09-10 23:43:14,710] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0157a6f4c941447eae72408497838ee2job_title%27
[2022-09-10 23:43:14,720] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.69703ms
[2022-09-10 23:43:15,198] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:43:15,373] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:43:18,969] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:43:19,725] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:43:19,758] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:43:19,785] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 23:43:19,789] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.904743ms
[2022-09-10 23:43:19,790] - [task:3246] INFO - {'requestID': '6fc3fd5f-4acc-447e-8da5-1490a4468946', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0157a6f4c941447eae72408497838ee2job_title', 'index_id': 'd937874c5f6c8451', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.904743ms', 'executionTime': '2.827388ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 23:43:19,790] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:43:19,790] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:43:19,791] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:43:19,791] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:43:19,792] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 23:43:19,792] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:43:19,792] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:43:19,793] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:43:19,793] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:43:20,791] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:43:20,822] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:43:20,850] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 23:43:21,075] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 209.974925ms
[2022-09-10 23:43:21,075] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:43:21,076] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:43:22,602] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:43:22,602] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:43:23,632] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0157a6f4c941447eae72408497838ee2job_title'
[2022-09-10 23:43:23,658] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0157a6f4c941447eae72408497838ee2job_title%27
[2022-09-10 23:43:23,667] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.974546ms
[2022-09-10 23:43:23,693] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0157a6f4c941447eae72408497838ee2job_title ON default USING GSI
[2022-09-10 23:43:23,719] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0157a6f4c941447eae72408497838ee2job_title+ON+default+USING+GSI
[2022-09-10 23:43:23,760] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.452272ms
[2022-09-10 23:43:23,800] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0157a6f4c941447eae72408497838ee2job_title'
[2022-09-10 23:43:23,826] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0157a6f4c941447eae72408497838ee2job_title%27
[2022-09-10 23:43:23,833] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.514509ms
[2022-09-10 23:43:23,833] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0d430ebb-8c03-4547-af87-85012a183d7b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.514509ms', 'executionTime': '5.456435ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:43:23,939] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:43:23,944] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:23,944] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:24,454] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:24,509] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:43:24,510] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:43:24,598] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:43:24,657] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:43:24,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:24,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:25,217] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:25,276] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:43:25,277] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:43:25,373] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:43:25,429] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:43:25,429] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:43:25,430] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:43:25,456] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:43:25,481] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:43:25,491] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.449931ms
[2022-09-10 23:43:25,519] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:43:25,546] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:43:25,606] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.658875ms
[2022-09-10 23:43:25,675] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:43:25,675] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.80054004450284, 'mem_free': 12944949248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:43:25,675] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:43:25,679] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:25,679] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:26,233] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:26,238] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:26,239] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:26,779] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:26,784] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:26,784] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:27,615] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:27,628] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:27,628] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:28,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:34,742] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:43:34,891] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:43:35,328] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:43:35,358] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:43:35,358] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:43:35,420] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:43:35,476] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:43:35,530] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:43:35,531] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:43:35,613] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:43:35,614] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:43:35,641] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:43:35,772] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:43:35,772] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:43:35,799] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:43:35,826] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:43:35,826] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:43:35,854] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:43:35,882] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:43:35,882] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:43:35,911] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:43:35,938] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:43:35,939] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:43:35,965] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:43:35,966] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:43:35,966] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:43:35,966] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 87.889s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_6'}
[2022-09-10 23:43:36,079] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:36,080] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:36,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:36,608] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:43:36,685] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:43:36,686] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-10 23:43:36,686] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:43:37,106] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:43:37,135] - [task:165] INFO -  {'uptime': '493', 'memoryTotal': 15466930176, 'memoryFree': 13035782144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:43:37,165] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:43:37,165] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:43:37,165] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:43:37,199] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:43:37,237] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:43:37,238] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:43:37,267] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:43:37,268] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:43:37,268] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:43:37,269] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:43:37,318] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:43:37,321] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:37,321] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:37,861] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:37,862] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:43:37,972] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:43:37,973] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:43:38,005] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:38,033] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:38,064] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:43:38,187] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:43:38,215] - [task:165] INFO -  {'uptime': '490', 'memoryTotal': 15466930176, 'memoryFree': 13034999808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:43:38,242] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:43:38,273] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:43:38,273] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:43:38,332] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:43:38,335] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:38,335] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:38,888] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:38,889] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:43:39,001] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:43:39,002] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:43:39,034] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:39,061] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:39,092] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:43:39,215] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:43:39,242] - [task:165] INFO -  {'uptime': '492', 'memoryTotal': 15466930176, 'memoryFree': 13034868736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:43:39,268] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:43:39,297] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:43:39,298] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:43:39,348] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:43:39,351] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:39,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:39,854] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:39,855] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:43:39,973] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:43:39,974] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:43:40,004] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:40,030] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:40,059] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:43:40,172] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:43:40,197] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13035397120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:43:40,222] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:43:40,250] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:43:40,251] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:43:40,301] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:43:40,305] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:40,305] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:40,856] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:40,857] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:43:40,960] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:43:40,961] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:43:40,990] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:41,016] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:43:41,044] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:43:41,135] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:43:41,688] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:43:46,689] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:43:46,770] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:43:46,775] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:46,775] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:47,305] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:47,306] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:43:47,415] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:43:47,417] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:43:47,417] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:43:48,248] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:43:48,301] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 23:43:48,302] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:43:49,219] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:43:49,534] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:43:49,835] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:43:49,839] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:43:49,905] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:49,905] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:50,406] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:50,411] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:50,412] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:50,960] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:50,971] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:50,971] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:51,879] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:51,888] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:43:51,888] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:43:52,802] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:43:58,644] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:43:58,644] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.80532202593771, 'mem_free': 13038010368, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:43:58,644] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:43:58,645] - [basetestcase:467] INFO - Time to execute basesetup : 22.567384958267212
[2022-09-10 23:43:58,696] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:43:58,697] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:43:58,751] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:43:58,751] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:43:58,804] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:43:58,804] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:43:58,856] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:43:58,856] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:43:58,908] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:43:58,967] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:43:58,967] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:43:58,967] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:44:03,976] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:44:03,981] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:44:03,981] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:44:04,528] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:44:05,553] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:44:05,812] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:44:08,025] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:44:08,042] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 23:44:09,414] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:44:09,414] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:44:09,415] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:44:39,432] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:44:39,462] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:44:39,489] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:44:39,560] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.930975ms
[2022-09-10 23:44:39,560] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '29510d87-871d-41a6-801d-8839ffc7ab15', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.930975ms', 'executionTime': '68.857388ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:44:39,560] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:44:39,588] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:44:39,613] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:44:40,300] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 685.643286ms
[2022-09-10 23:44:40,301] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:44:40,363] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:44:40,403] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:44:40,411] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.268195ms
[2022-09-10 23:44:40,603] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:44:40,656] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:44:40,671] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:44:40,671] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:44:40,688] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:44:40,757] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:44:41,571] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee1677febf98f6445bbb52caee900bc3d5job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:44:41,597] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee1677febf98f6445bbb52caee900bc3d5job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:44:41,675] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 73.411543ms
[2022-09-10 23:44:41,676] - [base_gsi:282] INFO - BUILD INDEX on default(employee1677febf98f6445bbb52caee900bc3d5job_title) USING GSI
[2022-09-10 23:44:42,706] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee1677febf98f6445bbb52caee900bc3d5job_title) USING GSI
[2022-09-10 23:44:42,733] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee1677febf98f6445bbb52caee900bc3d5job_title%29+USING+GSI
[2022-09-10 23:44:42,769] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.157123ms
[2022-09-10 23:44:43,800] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1677febf98f6445bbb52caee900bc3d5job_title'
[2022-09-10 23:44:43,828] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1677febf98f6445bbb52caee900bc3d5job_title%27
[2022-09-10 23:44:43,838] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.159717ms
[2022-09-10 23:44:44,868] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1677febf98f6445bbb52caee900bc3d5job_title'
[2022-09-10 23:44:44,895] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1677febf98f6445bbb52caee900bc3d5job_title%27
[2022-09-10 23:44:44,905] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.815389ms
[2022-09-10 23:44:45,415] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 23:44:45,587] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:44:46,705] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:44:47,325] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 23:44:47,501] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:44:48,814] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:44:49,206] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:44:49,383] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:44:53,055] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:44:53,923] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:44:53,958] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:44:53,987] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 23:44:53,992] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.885621ms
[2022-09-10 23:44:53,992] - [task:3246] INFO - {'requestID': 'd67e4ad9-473f-411b-b6aa-7b3f93443c4f', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee1677febf98f6445bbb52caee900bc3d5job_title', 'index_id': '42f6c3e2d1abaf56', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.885621ms', 'executionTime': '2.811295ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 23:44:53,992] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:44:53,993] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:44:53,993] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:44:53,993] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:44:53,994] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 23:44:53,994] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:44:53,994] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:44:53,995] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:44:53,995] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 23:44:54,994] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:44:55,025] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 23:44:55,056] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 23:44:55,448] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 168.158727ms
[2022-09-10 23:44:55,449] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:44:55,450] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:44:57,026] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:44:57,026] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 23:44:58,058] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1677febf98f6445bbb52caee900bc3d5job_title'
[2022-09-10 23:44:58,084] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1677febf98f6445bbb52caee900bc3d5job_title%27
[2022-09-10 23:44:58,095] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.87447ms
[2022-09-10 23:44:58,122] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee1677febf98f6445bbb52caee900bc3d5job_title ON default USING GSI
[2022-09-10 23:44:58,148] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee1677febf98f6445bbb52caee900bc3d5job_title+ON+default+USING+GSI
[2022-09-10 23:44:58,217] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.443444ms
[2022-09-10 23:44:58,249] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1677febf98f6445bbb52caee900bc3d5job_title'
[2022-09-10 23:44:58,278] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1677febf98f6445bbb52caee900bc3d5job_title%27
[2022-09-10 23:44:58,288] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.592711ms
[2022-09-10 23:44:58,289] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '66811671-96f2-471d-8065-350db5ed2ef4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.592711ms', 'executionTime': '7.501945ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:44:58,403] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:44:58,405] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:44:58,406] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:44:59,120] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:44:59,179] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:44:59,179] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:44:59,289] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:44:59,349] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:44:59,352] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:44:59,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:00,037] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:00,098] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:45:00,099] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:45:00,207] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:45:00,268] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:45:00,269] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:45:00,269] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:45:00,295] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:45:00,324] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:45:00,333] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.700924ms
[2022-09-10 23:45:00,360] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:45:00,386] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:45:00,427] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.514462ms
[2022-09-10 23:45:00,498] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:45:00,498] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.92809767641106, 'mem_free': 12817465344, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:45:00,498] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:45:00,503] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:00,503] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:01,201] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:01,205] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:01,206] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:01,890] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:01,895] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:01,895] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:03,048] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:03,056] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:03,056] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:04,204] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:11,563] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:45:11,709] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:45:12,860] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:45:12,889] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:45:12,890] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:45:12,954] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:45:13,012] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:45:13,065] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:45:13,066] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:45:13,148] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:45:13,150] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:45:13,176] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:45:13,316] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:45:13,316] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:45:13,345] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:45:13,371] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:45:13,371] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:45:13,398] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:45:13,424] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:45:13,424] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:45:13,451] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:45:13,477] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:45:13,477] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:45:13,504] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:45:13,504] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 23:45:13,504] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:45:13,505] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 97.479s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_7'}
[2022-09-10 23:45:13,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:13,639] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:14,245] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:14,278] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:45:14,356] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:45:14,356] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-10 23:45:14,357] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:45:14,666] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:45:14,693] - [task:165] INFO -  {'uptime': '591', 'memoryTotal': 15466930176, 'memoryFree': 12980469760, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:45:14,718] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:45:14,719] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:45:14,719] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:45:14,753] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:45:14,791] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:45:14,791] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:45:14,820] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:45:14,820] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:45:14,821] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:45:14,821] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:45:14,869] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:45:14,872] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:14,872] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:15,466] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:15,468] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:45:15,582] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:45:15,584] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:45:15,612] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:15,637] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:15,666] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:45:15,784] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:45:15,814] - [task:165] INFO -  {'uptime': '591', 'memoryTotal': 15466930176, 'memoryFree': 12970340352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:45:15,840] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:45:15,867] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:45:15,867] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:45:15,920] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:45:15,923] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:15,923] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:16,502] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:16,503] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:45:16,615] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:45:16,617] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:45:16,646] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:16,671] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:16,699] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:45:16,817] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:45:16,846] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12979523584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:45:16,871] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:45:16,899] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:45:16,899] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:45:16,950] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:45:16,955] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:16,955] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:17,549] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:17,550] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:45:17,662] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:45:17,662] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:45:17,692] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:17,717] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:17,745] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:45:17,860] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:45:17,890] - [task:165] INFO -  {'uptime': '588', 'memoryTotal': 15466930176, 'memoryFree': 12995596288, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:45:17,916] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:45:17,944] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:45:17,944] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:45:17,994] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:45:17,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:17,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:18,563] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:18,564] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:45:18,677] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:45:18,678] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:45:18,711] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:18,738] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:45:18,767] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:45:18,853] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:45:19,537] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:45:24,542] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:45:24,626] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:45:24,630] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:24,630] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:25,220] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:25,221] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:45:25,335] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:45:25,336] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:45:25,337] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:45:25,988] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:45:26,042] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 23:45:26,042] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:45:26,976] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:45:27,307] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:45:27,576] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:45:27,579] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 23:45:27,638] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:27,638] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:28,243] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:28,248] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:28,248] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:29,023] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:29,035] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:29,035] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:30,113] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:30,126] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:30,126] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:31,184] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:37,546] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:45:37,547] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.98448380336052, 'mem_free': 12991770624, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:45:37,547] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:45:37,547] - [basetestcase:467] INFO - Time to execute basesetup : 23.91076636314392
[2022-09-10 23:45:37,598] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:45:37,599] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:45:37,652] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:45:37,652] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:45:37,706] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:45:37,706] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:45:37,760] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:45:37,760] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:45:37,811] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:45:37,868] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:45:37,868] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:45:37,869] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:45:42,881] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:45:42,886] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:45:42,886] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:45:43,520] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:45:44,565] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:45:44,739] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:45:46,946] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:45:47,025] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:45:47,025] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:45:47,026] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:46:17,053] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:46:17,082] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:46:17,108] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:46:17,180] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.305517ms
[2022-09-10 23:46:17,181] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2391134d-c85a-488a-b2c9-7d729b28ebca', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.305517ms', 'executionTime': '70.204861ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:46:17,181] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:46:17,207] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:46:17,235] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:46:17,945] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 708.59096ms
[2022-09-10 23:46:17,946] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:46:18,033] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:46:18,061] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:46:18,073] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.508015ms
[2022-09-10 23:46:18,273] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:46:18,308] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:46:18,327] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:46:18,328] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:46:18,347] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:46:18,347] - [base_gsi:326] INFO - []
[2022-09-10 23:46:19,237] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0af5f2b306f74899ab839d96956e9305job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:46:19,262] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0af5f2b306f74899ab839d96956e9305job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:46:19,320] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.72601ms
[2022-09-10 23:46:19,357] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0af5f2b306f74899ab839d96956e9305join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:46:19,384] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0af5f2b306f74899ab839d96956e9305join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:46:19,439] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.816734ms
[2022-09-10 23:46:19,439] - [base_gsi:282] INFO - BUILD INDEX on default(employee0af5f2b306f74899ab839d96956e9305job_title,employee0af5f2b306f74899ab839d96956e9305join_yr) USING GSI
[2022-09-10 23:46:20,470] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee0af5f2b306f74899ab839d96956e9305job_title,employee0af5f2b306f74899ab839d96956e9305join_yr) USING GSI
[2022-09-10 23:46:20,496] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee0af5f2b306f74899ab839d96956e9305job_title%2Cemployee0af5f2b306f74899ab839d96956e9305join_yr%29+USING+GSI
[2022-09-10 23:46:20,538] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.339368ms
[2022-09-10 23:46:21,569] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0af5f2b306f74899ab839d96956e9305job_title'
[2022-09-10 23:46:21,595] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0af5f2b306f74899ab839d96956e9305job_title%27
[2022-09-10 23:46:21,602] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.928845ms
[2022-09-10 23:46:22,633] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0af5f2b306f74899ab839d96956e9305job_title'
[2022-09-10 23:46:22,658] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0af5f2b306f74899ab839d96956e9305job_title%27
[2022-09-10 23:46:22,665] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.387593ms
[2022-09-10 23:46:22,691] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0af5f2b306f74899ab839d96956e9305join_yr'
[2022-09-10 23:46:22,717] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0af5f2b306f74899ab839d96956e9305join_yr%27
[2022-09-10 23:46:22,720] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.438476ms
[2022-09-10 23:46:23,751] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0af5f2b306f74899ab839d96956e9305job_title'
[2022-09-10 23:46:23,778] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0af5f2b306f74899ab839d96956e9305job_title%27
[2022-09-10 23:46:23,788] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.195393ms
[2022-09-10 23:46:23,816] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0af5f2b306f74899ab839d96956e9305job_title ON default USING GSI
[2022-09-10 23:46:23,843] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0af5f2b306f74899ab839d96956e9305job_title+ON+default+USING+GSI
[2022-09-10 23:46:23,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.039017ms
[2022-09-10 23:46:23,927] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0af5f2b306f74899ab839d96956e9305join_yr'
[2022-09-10 23:46:23,958] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0af5f2b306f74899ab839d96956e9305join_yr%27
[2022-09-10 23:46:23,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.032353ms
[2022-09-10 23:46:23,996] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0af5f2b306f74899ab839d96956e9305join_yr ON default USING GSI
[2022-09-10 23:46:24,024] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0af5f2b306f74899ab839d96956e9305join_yr+ON+default+USING+GSI
[2022-09-10 23:46:24,070] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.335296ms
[2022-09-10 23:46:24,104] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0af5f2b306f74899ab839d96956e9305job_title'
[2022-09-10 23:46:24,132] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0af5f2b306f74899ab839d96956e9305job_title%27
[2022-09-10 23:46:24,141] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.086976ms
[2022-09-10 23:46:24,142] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9c782533-1dd0-4dcf-8c31-5611f5070a9a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.086976ms', 'executionTime': '7.001057ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:46:24,173] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0af5f2b306f74899ab839d96956e9305join_yr'
[2022-09-10 23:46:24,199] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0af5f2b306f74899ab839d96956e9305join_yr%27
[2022-09-10 23:46:24,202] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.488525ms
[2022-09-10 23:46:24,203] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a38310c6-b55b-48fa-8e40-b9f6c2648132', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.488525ms', 'executionTime': '1.428987ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:46:24,308] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:46:24,311] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:24,311] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:24,969] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:25,024] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:46:25,025] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:46:25,128] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:46:25,187] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:46:25,190] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:25,191] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:25,805] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:25,861] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:46:25,861] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:46:25,965] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:46:26,025] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:46:26,026] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:46:26,026] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:46:26,052] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:46:26,079] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:46:26,088] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.126481ms
[2022-09-10 23:46:26,119] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:46:26,144] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:46:26,198] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.585298ms
[2022-09-10 23:46:26,274] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:46:26,274] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.170311462558097, 'mem_free': 12876374016, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:46:26,275] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:46:26,279] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:26,280] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:26,900] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:26,907] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:26,907] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:27,547] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:27,552] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:27,552] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:28,555] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:28,570] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:28,570] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:29,657] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:36,734] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-10 23:46:36,879] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:46:37,521] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:46:37,550] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:46:37,550] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:46:37,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:46:37,663] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:46:37,718] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:46:37,719] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:46:37,801] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:46:37,802] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:46:37,829] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:46:37,964] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:46:37,964] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:46:38,001] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:46:38,035] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:46:38,035] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:46:38,061] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:46:38,087] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:46:38,087] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:46:38,114] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:46:38,139] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:46:38,139] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:46:38,165] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:46:38,165] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 23:46:38,165] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:46:38,166] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.581s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_8'}
[2022-09-10 23:46:38,245] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:38,245] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:38,835] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:38,868] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:46:38,947] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:46:38,947] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-10 23:46:38,948] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:46:39,272] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:46:39,298] - [task:165] INFO -  {'uptime': '675', 'memoryTotal': 15466930176, 'memoryFree': 12997308416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:46:39,323] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:46:39,324] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:46:39,324] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:46:39,357] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:46:39,397] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:46:39,397] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:46:39,425] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:46:39,426] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:46:39,426] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:46:39,426] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:46:39,473] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:46:39,476] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:39,476] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:40,069] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:40,070] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:46:40,183] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:46:40,184] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:46:40,213] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:40,239] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:40,274] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:46:40,397] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:46:40,424] - [task:165] INFO -  {'uptime': '671', 'memoryTotal': 15466930176, 'memoryFree': 12996476928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:46:40,450] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:46:40,480] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:46:40,480] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:46:40,534] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:46:40,537] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:40,537] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:41,141] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:41,142] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:46:41,267] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:46:41,268] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:46:41,297] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:41,323] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:41,352] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:46:41,485] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:46:41,511] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 12997083136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:46:41,536] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:46:41,564] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:46:41,564] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:46:41,614] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:46:41,619] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:41,619] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:42,212] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:42,213] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:46:42,326] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:46:42,327] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:46:42,357] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:42,383] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:42,412] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:46:42,534] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:46:42,563] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 12996878336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:46:42,589] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:46:42,616] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:46:42,617] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:46:42,668] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:46:42,671] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:42,671] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:43,294] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:43,296] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:46:43,413] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:46:43,414] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:46:43,444] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:43,471] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:46:43,499] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:46:43,587] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:46:44,113] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:46:49,114] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:46:49,204] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:46:49,209] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:49,209] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:49,865] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:49,866] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:46:49,988] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:46:49,989] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:46:49,989] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:46:50,768] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:46:50,825] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 23:46:50,826] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:46:51,713] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:46:52,024] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:46:52,422] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:46:52,425] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 23:46:52,478] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:52,478] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:53,079] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:53,085] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:53,085] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:53,955] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:53,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:53,967] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:55,051] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:46:55,060] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:46:55,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:46:56,154] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:47:02,834] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:47:02,834] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 28.28469488976353, 'mem_free': 12993363968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:47:02,834] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:47:02,835] - [basetestcase:467] INFO - Time to execute basesetup : 24.59216070175171
[2022-09-10 23:47:02,888] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:47:02,888] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:47:02,942] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:47:02,942] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:47:02,997] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:47:02,997] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:47:03,051] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:47:03,051] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:47:03,102] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:47:03,159] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:47:03,160] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:47:03,160] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:47:08,172] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:47:08,175] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:47:08,175] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:47:08,786] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:47:09,819] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:47:09,991] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:47:12,219] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:47:12,304] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:47:12,304] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:47:12,305] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:47:42,332] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:47:42,361] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:47:42,387] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:47:42,455] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.896229ms
[2022-09-10 23:47:42,455] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5b07381b-4506-4ea9-9ab5-44b5450c7d4a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.896229ms', 'executionTime': '65.835663ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:47:42,455] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:47:42,482] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:47:42,508] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:47:43,201] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 690.540873ms
[2022-09-10 23:47:43,201] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:47:43,272] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:47:43,312] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:47:43,319] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.800008ms
[2022-09-10 23:47:43,515] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:47:43,550] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:47:43,563] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:47:43,564] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:47:43,582] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:47:43,583] - [base_gsi:326] INFO - []
[2022-09-10 23:47:44,487] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:47:44,513] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:47:44,575] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.391534ms
[2022-09-10 23:47:44,576] - [base_gsi:282] INFO - BUILD INDEX on default(employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr) USING GSI
[2022-09-10 23:47:45,607] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr) USING GSI
[2022-09-10 23:47:45,632] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr%29+USING+GSI
[2022-09-10 23:47:45,653] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.275397ms
[2022-09-10 23:47:46,684] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr'
[2022-09-10 23:47:46,709] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr%27
[2022-09-10 23:47:46,716] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.653357ms
[2022-09-10 23:47:47,746] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr'
[2022-09-10 23:47:47,772] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr%27
[2022-09-10 23:47:47,779] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.503445ms
[2022-09-10 23:47:48,811] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr'
[2022-09-10 23:47:48,837] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr%27
[2022-09-10 23:47:48,847] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.829611ms
[2022-09-10 23:47:48,873] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr ON default USING GSI
[2022-09-10 23:47:48,899] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr+ON+default+USING+GSI
[2022-09-10 23:47:48,936] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.14627ms
[2022-09-10 23:47:48,974] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr'
[2022-09-10 23:47:49,004] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeebc2c8d5b91e4e35b3e0768b8ce2e9e4job_title_join_yr%27
[2022-09-10 23:47:49,011] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.262489ms
[2022-09-10 23:47:49,012] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '33e48aa7-6c17-4041-9bd1-0bbe01ababba', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.262489ms', 'executionTime': '5.180076ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:47:49,113] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:47:49,116] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:47:49,117] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:47:49,759] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:47:49,815] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:47:49,816] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:47:49,923] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:47:49,982] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:47:49,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:47:49,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:47:50,618] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:47:50,675] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:47:50,675] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:47:50,781] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:47:50,845] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:47:50,846] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:47:50,846] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:47:50,872] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:47:50,902] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:47:50,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.87434ms
[2022-09-10 23:47:50,939] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:47:50,964] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:47:51,028] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.353137ms
[2022-09-10 23:47:51,099] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:47:51,099] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.751417542526276, 'mem_free': 12872966144, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:47:51,100] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:47:51,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:47:51,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:47:51,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:47:51,776] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:47:51,776] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:47:52,833] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:47:52,841] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:47:52,841] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:47:53,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:47:53,968] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:47:53,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:47:55,082] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:01,695] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-10 23:48:01,841] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:48:02,365] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:48:02,392] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:48:02,392] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:48:02,447] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:48:02,500] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:48:02,551] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:48:02,552] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:48:02,630] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:48:02,631] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:48:02,656] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:48:02,786] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:48:02,786] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:48:02,812] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:48:02,839] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:48:02,839] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:48:02,866] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:48:02,891] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:48:02,892] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:48:02,919] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:48:02,944] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:48:02,944] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:48:02,977] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:48:02,977] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 23:48:02,977] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:48:02,978] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.792s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_9'}
[2022-09-10 23:48:03,066] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:03,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:03,673] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:03,706] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:48:03,787] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:48:03,787] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-10 23:48:03,788] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:48:04,092] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:48:04,119] - [task:165] INFO -  {'uptime': '760', 'memoryTotal': 15466930176, 'memoryFree': 12985466880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:48:04,145] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:48:04,145] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:48:04,145] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:48:04,182] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:48:04,216] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:48:04,216] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:48:04,244] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:48:04,244] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:48:04,245] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:48:04,245] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:48:04,293] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:48:04,296] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:04,296] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:04,933] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:04,935] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:48:05,059] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:48:05,060] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:48:05,091] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:05,118] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:05,147] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:48:05,273] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:48:05,301] - [task:165] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 12991000576, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:48:05,328] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:48:05,360] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:48:05,360] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:48:05,413] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:48:05,416] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:05,416] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:06,072] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:06,073] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:48:06,201] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:48:06,202] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:48:06,233] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:06,260] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:06,288] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:48:06,404] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:48:06,433] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 12980404224, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:48:06,460] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:48:06,490] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:48:06,490] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:48:06,543] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:48:06,549] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:06,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:07,184] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:07,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:48:07,304] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:48:07,305] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:48:07,335] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:07,361] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:07,393] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:48:07,511] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:48:07,538] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 12989177856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:48:07,563] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:48:07,590] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:48:07,591] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:48:07,641] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:48:07,644] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:07,644] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:08,306] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:08,307] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:48:08,437] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:48:08,438] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:48:08,469] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:08,496] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:48:08,526] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:48:08,618] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:48:09,142] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:48:14,146] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:48:14,231] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:48:14,236] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:14,236] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:14,869] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:14,870] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:48:14,989] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:48:14,990] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:48:14,991] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:48:15,788] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:48:15,844] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 23:48:15,845] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:48:16,881] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:48:17,188] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:48:17,484] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:48:17,487] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 23:48:17,554] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:17,555] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:18,206] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:18,212] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:18,212] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:19,174] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:19,186] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:19,186] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:20,296] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:20,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:20,309] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:21,415] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:27,974] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:48:27,974] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.59609839024703, 'mem_free': 12990525440, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:48:27,975] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:48:27,975] - [basetestcase:467] INFO - Time to execute basesetup : 24.91154146194458
[2022-09-10 23:48:28,025] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:48:28,025] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:48:28,079] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:48:28,079] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:48:28,133] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:48:28,134] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:48:28,187] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:48:28,187] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:48:28,239] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:48:28,300] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:48:28,300] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:48:28,301] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:48:33,308] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:48:33,312] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:48:33,312] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:48:33,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:48:35,109] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:48:35,277] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:48:37,392] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:48:37,473] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:48:37,474] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:48:37,474] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:49:07,504] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:49:07,534] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:49:07,559] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:49:07,626] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.00086ms
[2022-09-10 23:49:07,626] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7bee9ba4-afb9-4a67-a092-1f483da302e2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.00086ms', 'executionTime': '64.942356ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:49:07,626] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:49:07,652] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:49:07,676] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:49:08,330] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 651.283585ms
[2022-09-10 23:49:08,330] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:49:08,380] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:49:08,420] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:49:08,428] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.38623ms
[2022-09-10 23:49:08,626] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:49:08,661] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:49:08,679] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:49:08,679] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:49:08,700] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:49:08,774] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:49:09,591] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee1b84328308549fc8e908a3f8765b8fdjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:49:09,617] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee1b84328308549fc8e908a3f8765b8fdjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:49:09,674] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.693121ms
[2022-09-10 23:49:09,676] - [base_gsi:282] INFO - BUILD INDEX on default(`employeee1b84328308549fc8e908a3f8765b8fdjoin_yr`) USING GSI
[2022-09-10 23:49:10,707] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeee1b84328308549fc8e908a3f8765b8fdjoin_yr`) USING GSI
[2022-09-10 23:49:10,732] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeee1b84328308549fc8e908a3f8765b8fdjoin_yr%60%29+USING+GSI
[2022-09-10 23:49:10,753] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.788208ms
[2022-09-10 23:49:10,795] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1b84328308549fc8e908a3f8765b8fdjoin_yr'
[2022-09-10 23:49:10,824] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1b84328308549fc8e908a3f8765b8fdjoin_yr%27
[2022-09-10 23:49:10,834] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.086127ms
[2022-09-10 23:49:11,868] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1b84328308549fc8e908a3f8765b8fdjoin_yr'
[2022-09-10 23:49:11,894] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1b84328308549fc8e908a3f8765b8fdjoin_yr%27
[2022-09-10 23:49:11,901] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.888118ms
[2022-09-10 23:49:12,931] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1b84328308549fc8e908a3f8765b8fdjoin_yr'
[2022-09-10 23:49:12,957] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1b84328308549fc8e908a3f8765b8fdjoin_yr%27
[2022-09-10 23:49:12,964] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.744404ms
[2022-09-10 23:49:13,994] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:49:14,020] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 23:49:14,024] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.823943ms
[2022-09-10 23:49:14,024] - [base_gsi:504] INFO - {'requestID': '01bf30ec-eaf0-4f2f-8175-4c3e3b10e7f3', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee1b84328308549fc8e908a3f8765b8fdjoin_yr', 'index_id': 'e2de71d8405a89cd', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.823943ms', 'executionTime': '2.755958ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 23:49:14,025] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:49:14,025] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:49:14,025] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:49:14,026] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:49:14,027] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:49:14,027] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:49:14,027] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:49:14,028] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:49:14,090] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:49:14,132] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:49:14,159] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 23:49:14,335] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 163.677443ms
[2022-09-10 23:49:14,335] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:49:14,336] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:49:16,384] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:49:16,409] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1b84328308549fc8e908a3f8765b8fdjoin_yr'
[2022-09-10 23:49:16,435] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1b84328308549fc8e908a3f8765b8fdjoin_yr%27
[2022-09-10 23:49:16,440] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.003203ms
[2022-09-10 23:49:16,440] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '56aeaa42-b1c9-4344-9e99-c54c77be3b1b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.003203ms', 'executionTime': '3.947105ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:49:16,542] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:49:16,545] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:16,545] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:17,249] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:17,308] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:49:17,309] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:49:17,419] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:49:17,477] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:49:17,480] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:17,480] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:18,110] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:18,166] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:49:18,166] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:49:18,272] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:49:18,328] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:49:18,328] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:49:18,329] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:49:18,354] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:49:18,379] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:49:18,384] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.556543ms
[2022-09-10 23:49:18,385] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '13eb3c60-75d0-497c-bd7e-ff1ce84776e4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.556543ms', 'executionTime': '3.479817ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:49:18,437] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:49:18,437] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.630949160701727, 'mem_free': 12852486144, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:49:18,438] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:49:18,441] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:18,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:19,146] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:19,151] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:19,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:19,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:19,842] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:19,842] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:20,900] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:20,909] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:20,909] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:22,098] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:29,547] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-10 23:49:29,681] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:49:29,734] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:49:29,790] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:49:29,843] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:49:29,843] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:49:29,920] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:49:29,921] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:49:29,945] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:49:30,075] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:49:30,075] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:49:30,101] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:49:30,126] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:49:30,126] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:49:30,153] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:49:30,178] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:49:30,178] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:49:30,204] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:49:30,229] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:49:30,229] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:49:30,255] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:49:30,255] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 23:49:30,255] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:49:30,256] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 87.244s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_10'}
[2022-09-10 23:49:30,337] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:30,337] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:31,002] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:31,036] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:49:31,121] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:49:31,121] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-10 23:49:31,122] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:49:31,365] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:49:31,397] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 12831502336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:49:31,425] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:49:31,426] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:49:31,426] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:49:31,456] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:49:31,492] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:49:31,493] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:49:31,523] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:49:31,524] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:49:31,524] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:49:31,524] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:49:31,573] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:49:31,575] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:31,576] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:32,234] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:32,235] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:49:32,356] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:49:32,357] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:49:32,387] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:32,412] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:32,441] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:49:32,564] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:49:32,590] - [task:165] INFO -  {'uptime': '847', 'memoryTotal': 15466930176, 'memoryFree': 12977074176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:49:32,616] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:49:32,643] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:49:32,643] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:49:32,695] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:49:32,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:32,700] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:33,331] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:33,332] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:49:33,454] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:49:33,455] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:49:33,486] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:33,511] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:33,539] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:49:33,653] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:49:33,680] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 12977156096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:49:33,705] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:49:33,732] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:49:33,732] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:49:33,782] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:49:33,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:33,786] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:34,412] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:34,413] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:49:34,539] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:49:34,541] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:49:34,572] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:34,598] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:34,627] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:49:34,745] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:49:34,772] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 12977508352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:49:34,799] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:49:34,830] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:49:34,830] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:49:34,884] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:49:34,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:34,887] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:35,518] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:35,520] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:49:35,639] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:49:35,641] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:49:35,671] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:35,696] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:49:35,724] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:49:35,809] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:49:36,406] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:49:41,410] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:49:41,497] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:49:41,502] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:41,502] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:42,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:42,176] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:49:42,306] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:49:42,307] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:49:42,307] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:49:42,984] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:49:43,044] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 23:49:43,044] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:49:43,948] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:49:44,261] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:49:44,538] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:49:44,541] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 23:49:44,609] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:44,609] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:45,279] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:45,284] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:45,284] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:46,259] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:46,268] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:46,268] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:47,414] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:47,427] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:49:47,427] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:49:48,558] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:49:55,459] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:49:55,459] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.17206253746843, 'mem_free': 12971999232, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:49:55,459] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:49:55,460] - [basetestcase:467] INFO - Time to execute basesetup : 25.125032901763916
[2022-09-10 23:49:55,510] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:49:55,510] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:49:55,562] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:49:55,562] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:49:55,615] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:49:55,615] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:49:55,666] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:49:55,666] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:49:55,716] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:49:55,775] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:49:55,775] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:49:55,775] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:50:00,787] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:50:00,790] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:50:00,790] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:50:01,499] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:50:02,683] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:50:02,860] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:50:05,911] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:50:05,989] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:50:05,989] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:50:05,989] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:50:36,017] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:50:36,048] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:50:36,074] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:50:36,147] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.834641ms
[2022-09-10 23:50:36,147] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '96c879a4-cd63-4633-8070-6feaf2622b0c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.834641ms', 'executionTime': '70.769896ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:50:36,147] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:50:36,176] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:50:36,202] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:50:36,893] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 688.655264ms
[2022-09-10 23:50:36,893] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:50:36,964] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:50:37,025] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:50:37,035] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.536946ms
[2022-09-10 23:50:37,226] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:50:37,261] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:50:37,281] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:50:37,282] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:50:37,296] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:50:37,372] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:50:38,190] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee87288c4c6b034e72a49ea7cde722c495join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:50:38,215] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee87288c4c6b034e72a49ea7cde722c495join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:50:38,286] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.558747ms
[2022-09-10 23:50:38,286] - [base_gsi:282] INFO - BUILD INDEX on default(`employee87288c4c6b034e72a49ea7cde722c495join_yr`) USING GSI
[2022-09-10 23:50:39,317] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee87288c4c6b034e72a49ea7cde722c495join_yr`) USING GSI
[2022-09-10 23:50:39,343] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee87288c4c6b034e72a49ea7cde722c495join_yr%60%29+USING+GSI
[2022-09-10 23:50:39,367] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.590715ms
[2022-09-10 23:50:39,418] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87288c4c6b034e72a49ea7cde722c495join_yr'
[2022-09-10 23:50:39,456] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87288c4c6b034e72a49ea7cde722c495join_yr%27
[2022-09-10 23:50:39,465] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.882715ms
[2022-09-10 23:50:40,497] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87288c4c6b034e72a49ea7cde722c495join_yr'
[2022-09-10 23:50:40,524] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87288c4c6b034e72a49ea7cde722c495join_yr%27
[2022-09-10 23:50:40,535] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.811086ms
[2022-09-10 23:50:41,569] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87288c4c6b034e72a49ea7cde722c495join_yr'
[2022-09-10 23:50:41,596] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87288c4c6b034e72a49ea7cde722c495join_yr%27
[2022-09-10 23:50:41,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.781509ms
[2022-09-10 23:50:42,635] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:50:42,661] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 23:50:42,666] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.682554ms
[2022-09-10 23:50:42,666] - [base_gsi:504] INFO - {'requestID': '0a5435af-9a34-44a0-b0e0-00ebe4bb5676', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee87288c4c6b034e72a49ea7cde722c495join_yr', 'index_id': 'd7f00090586e96c9', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.682554ms', 'executionTime': '2.61305ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 23:50:42,666] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:50:42,667] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:50:42,667] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:50:42,667] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:50:42,668] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:50:42,668] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:50:42,668] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:50:42,668] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:50:42,729] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:50:42,771] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:50:42,798] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 23:50:42,969] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 159.638837ms
[2022-09-10 23:50:42,969] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:50:42,970] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:50:44,177] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-10 23:50:44,233] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-10 23:50:44,263] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:50:44,291] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 23:50:44,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.870969ms
[2022-09-10 23:50:44,294] - [base_gsi:504] INFO - {'requestID': 'd622d97b-0b85-47cf-87b1-b4d8d52a3dc3', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee87288c4c6b034e72a49ea7cde722c495join_yr', 'index_id': 'd7f00090586e96c9', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.870969ms', 'executionTime': '1.748743ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 23:50:44,295] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:50:44,295] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:50:44,295] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:50:44,295] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:50:44,295] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:50:44,296] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:50:44,296] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:50:44,296] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:50:44,348] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:50:44,387] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:50:44,413] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 23:50:44,520] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.421727ms
[2022-09-10 23:50:44,520] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:50:44,521] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:50:45,695] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee87288c4c6b034e72a49ea7cde722c495join_yr ON default USING GSI
[2022-09-10 23:50:45,721] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee87288c4c6b034e72a49ea7cde722c495join_yr+ON+default+USING+GSI
[2022-09-10 23:50:45,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.047315ms
[2022-09-10 23:50:45,794] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87288c4c6b034e72a49ea7cde722c495join_yr'
[2022-09-10 23:50:45,821] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87288c4c6b034e72a49ea7cde722c495join_yr%27
[2022-09-10 23:50:45,829] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.213757ms
[2022-09-10 23:50:45,830] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8bcc6c7b-b08d-47c2-aeae-a63745603b8a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.213757ms', 'executionTime': '6.140434ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:50:45,937] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:50:45,940] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:50:45,940] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:50:46,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:50:46,686] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:50:46,686] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:50:46,797] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:50:46,854] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:50:46,857] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:50:46,857] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:50:47,556] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:50:47,611] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:50:47,611] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:50:47,725] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:50:47,780] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:50:47,780] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:50:47,780] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:50:47,806] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:50:47,833] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:50:47,843] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.740903ms
[2022-09-10 23:50:47,868] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:50:47,894] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:50:47,951] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.614724ms
[2022-09-10 23:50:48,014] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:50:48,014] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 10.51608812051298, 'mem_free': 12806471680, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:50:48,014] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:50:48,018] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:50:48,018] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:50:48,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:50:48,695] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:50:48,695] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:50:49,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:50:49,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:50:49,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:50:50,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:50:50,345] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:50:50,345] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:50:51,629] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:50:59,560] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-10 23:50:59,712] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:51:00,516] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:51:00,545] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:51:00,546] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:51:00,604] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:51:00,659] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:51:00,716] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:51:00,717] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:51:00,800] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:51:00,801] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:51:00,830] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:51:00,974] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:51:00,974] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:51:01,002] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:51:01,030] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:51:01,030] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:51:01,058] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:51:01,085] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:51:01,085] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:51:01,113] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:51:01,141] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:51:01,142] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:51:01,169] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:51:01,169] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 23:51:01,169] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:51:01,170] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 90.889s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_11'}
[2022-09-10 23:51:01,255] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:01,255] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:01,991] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:02,026] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:51:02,106] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:51:02,106] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-10 23:51:02,107] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:51:02,282] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:51:02,309] - [task:165] INFO -  {'uptime': '938', 'memoryTotal': 15466930176, 'memoryFree': 12955697152, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:51:02,336] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:51:02,336] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:51:02,337] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:51:02,375] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:51:02,414] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:51:02,415] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:51:02,444] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:51:02,445] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:51:02,445] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:51:02,445] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:51:02,495] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:51:02,498] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:02,498] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:03,240] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:03,241] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:51:03,375] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:51:03,376] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:51:03,407] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:03,436] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:03,464] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:51:03,583] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:51:03,610] - [task:165] INFO -  {'uptime': '937', 'memoryTotal': 15466930176, 'memoryFree': 12954443776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:51:03,636] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:51:03,664] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:51:03,664] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:51:03,716] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:51:03,719] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:03,719] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:04,433] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:04,434] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:51:04,564] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:51:04,565] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:51:04,596] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:04,621] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:04,650] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:51:04,771] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:51:04,797] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12955189248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:51:04,823] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:51:04,852] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:51:04,853] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:51:04,906] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:51:04,911] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:04,911] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:05,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:05,592] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:51:05,720] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:51:05,722] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:51:05,751] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:05,778] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:05,806] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:51:05,925] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:51:05,952] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12956057600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:51:05,978] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:51:06,005] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:51:06,006] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:51:06,058] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:51:06,061] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:06,061] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:06,754] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:06,755] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:51:06,890] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:51:06,892] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:51:06,921] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:06,946] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:51:06,975] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:51:07,063] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:51:07,600] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:51:12,605] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:51:12,693] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:51:12,697] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:12,697] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:13,411] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:13,413] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:51:13,548] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:51:13,549] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:51:13,549] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:51:14,194] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:51:14,248] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 23:51:14,249] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:51:15,206] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:51:15,533] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:51:15,850] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:51:15,853] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 23:51:15,924] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:15,924] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:16,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:16,643] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:16,643] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:17,807] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:17,816] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:17,816] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:19,007] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:19,016] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:19,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:20,240] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:27,351] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:51:27,352] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.73141697533849, 'mem_free': 12955426816, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:51:27,352] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:51:27,352] - [basetestcase:467] INFO - Time to execute basesetup : 26.099933862686157
[2022-09-10 23:51:27,407] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:51:27,407] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:51:27,462] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:51:27,463] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:51:27,520] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:51:27,521] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:51:27,573] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:51:27,573] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:51:27,623] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:51:27,682] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:51:27,682] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:51:27,683] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:51:32,692] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:51:32,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:51:32,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:51:33,419] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:51:34,606] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:51:34,783] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:51:37,747] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:51:37,824] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:51:37,825] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:51:37,825] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:52:07,852] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:52:07,882] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:52:07,907] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:52:07,973] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.695886ms
[2022-09-10 23:52:07,973] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1c52ebc3-a0e8-4651-8e2b-3d6941b5d841', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.695886ms', 'executionTime': '63.624074ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:52:07,973] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:52:07,999] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:52:08,024] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:52:08,681] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 654.77898ms
[2022-09-10 23:52:08,682] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:52:08,741] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:52:08,794] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:52:08,802] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.071818ms
[2022-09-10 23:52:09,019] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:52:09,058] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:52:09,074] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:52:09,075] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:52:09,089] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:52:09,170] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:52:09,981] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeebb012f38846c43a39e103566a5ace694join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 23:52:10,006] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeebb012f38846c43a39e103566a5ace694join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 23:52:10,055] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.120968ms
[2022-09-10 23:52:10,056] - [base_gsi:282] INFO - BUILD INDEX on default(`employeebb012f38846c43a39e103566a5ace694join_yr`) USING GSI
[2022-09-10 23:52:11,088] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeebb012f38846c43a39e103566a5ace694join_yr`) USING GSI
[2022-09-10 23:52:11,115] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeebb012f38846c43a39e103566a5ace694join_yr%60%29+USING+GSI
[2022-09-10 23:52:11,135] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.771244ms
[2022-09-10 23:52:11,182] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebb012f38846c43a39e103566a5ace694join_yr'
[2022-09-10 23:52:11,226] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebb012f38846c43a39e103566a5ace694join_yr%27
[2022-09-10 23:52:11,235] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.36961ms
[2022-09-10 23:52:12,266] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebb012f38846c43a39e103566a5ace694join_yr'
[2022-09-10 23:52:12,291] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebb012f38846c43a39e103566a5ace694join_yr%27
[2022-09-10 23:52:12,300] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.873148ms
[2022-09-10 23:52:13,331] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebb012f38846c43a39e103566a5ace694join_yr'
[2022-09-10 23:52:13,357] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebb012f38846c43a39e103566a5ace694join_yr%27
[2022-09-10 23:52:13,367] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.697199ms
[2022-09-10 23:52:14,398] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:52:14,424] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 23:52:14,428] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.688063ms
[2022-09-10 23:52:14,428] - [base_gsi:504] INFO - {'requestID': '08f5152d-0f18-4f4c-8731-c642ccfb1e48', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeebb012f38846c43a39e103566a5ace694join_yr', 'index_id': 'bb3f70dc3c210fb', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.688063ms', 'executionTime': '2.622582ms', 'resultCount': 1, 'resultSize': 925, 'serviceLoad': 6}}
[2022-09-10 23:52:14,429] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 23:52:14,429] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:52:14,429] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 23:52:14,430] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 23:52:14,430] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:52:14,430] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:52:14,431] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:52:14,431] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 23:52:14,489] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 23:52:14,527] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 23:52:14,552] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 23:52:14,710] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 145.355984ms
[2022-09-10 23:52:14,710] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 23:52:14,710] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 23:52:16,767] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 23:52:19,506] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:52:19,564] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 23:52:19,564] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:52:20,448] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:52:20,774] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:52:21,059] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:52:21,062] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 23:52:23,093] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:52:23,094] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:23,120] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:52:23,147] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:52:23,148] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:23,175] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:52:23,203] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:52:23,203] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:23,231] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:52:23,259] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:52:23,259] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:23,288] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:52:23,338] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:52:23,363] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebb012f38846c43a39e103566a5ace694join_yr'
[2022-09-10 23:52:23,389] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebb012f38846c43a39e103566a5ace694join_yr%27
[2022-09-10 23:52:23,433] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.6345ms
[2022-09-10 23:52:23,433] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7647510d-628a-46b5-b8d8-d7ac9dec8ebd', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '42.6345ms', 'executionTime': '42.572865ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:52:23,460] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebb012f38846c43a39e103566a5ace694join_yr'
[2022-09-10 23:52:23,485] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebb012f38846c43a39e103566a5ace694join_yr%27
[2022-09-10 23:52:23,487] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.112959ms
[2022-09-10 23:52:23,488] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0db647ac-a84a-4c12-bcf8-c853eef56adb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.112959ms', 'executionTime': '1.063383ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:52:23,538] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:52:23,664] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:52:23,666] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:23,666] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:24,368] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:24,427] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:52:24,427] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:52:24,545] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:52:24,602] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:52:24,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:24,605] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:25,343] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:25,401] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:52:25,401] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:52:25,521] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:52:25,587] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:52:25,587] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:52:25,588] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:52:25,614] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:52:25,641] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:52:25,650] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.240069ms
[2022-09-10 23:52:25,650] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0c7076ef-3ab9-4383-8cf6-6c6514a7d169', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.240069ms', 'executionTime': '7.165261ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:52:25,650] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:52:25,676] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:52:25,702] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:52:25,704] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.08354ms
[2022-09-10 23:52:25,704] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b9386590-299e-4c15-8fc4-3005d112b164', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.08354ms', 'executionTime': '1.012538ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:52:25,756] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:52:25,756] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.53802957554753, 'mem_free': 12779098112, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:52:25,757] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:52:25,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:25,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:26,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:26,461] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:26,461] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:27,231] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:27,239] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:27,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:28,477] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:28,488] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:28,489] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:29,750] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:37,382] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 23:52:37,525] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:52:37,833] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:52:37,861] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:52:37,861] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:52:37,918] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:52:37,972] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:52:38,026] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:52:38,027] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:52:38,109] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:52:38,110] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:52:38,137] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:38,270] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:52:38,270] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:38,298] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:52:38,324] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:52:38,324] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:38,351] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:52:38,378] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:52:38,378] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:38,405] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:52:38,433] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:52:38,433] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:38,461] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:52:38,462] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 23:52:38,462] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:52:38,463] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 97.263s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_23-35-19/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-10 23:52:38,522] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:38,522] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:39,279] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:39,311] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:52:39,395] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 23:52:39,395] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-10 23:52:39,396] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 23:52:39,549] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 23:52:39,577] - [task:165] INFO -  {'uptime': '1036', 'memoryTotal': 15466930176, 'memoryFree': 12940763136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:52:39,604] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 23:52:39,604] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 23:52:39,604] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 23:52:39,643] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 23:52:39,677] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 23:52:39,677] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 23:52:39,706] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 23:52:39,707] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 23:52:39,708] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 23:52:39,708] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 23:52:39,758] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:52:39,761] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:39,761] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:40,476] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:40,477] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:52:40,611] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:52:40,612] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:52:40,647] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:40,673] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:40,701] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:52:40,826] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 23:52:40,853] - [task:165] INFO -  {'uptime': '1033', 'memoryTotal': 15466930176, 'memoryFree': 12942163968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:52:40,880] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:52:40,910] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 23:52:40,910] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 23:52:40,963] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:52:40,966] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:40,966] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:41,699] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:41,700] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:52:41,837] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:52:41,838] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:52:41,867] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:41,893] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:41,921] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:52:42,036] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 23:52:42,064] - [task:165] INFO -  {'uptime': '1034', 'memoryTotal': 15466930176, 'memoryFree': 12941021184, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:52:42,094] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:52:42,122] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 23:52:42,122] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 23:52:42,174] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:52:42,177] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:42,177] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:42,949] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:42,950] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:52:43,094] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:52:43,095] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:52:43,126] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:43,153] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:43,181] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:52:43,297] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 23:52:43,326] - [task:165] INFO -  {'uptime': '1035', 'memoryTotal': 15466930176, 'memoryFree': 12940767232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 23:52:43,352] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 23:52:43,380] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 23:52:43,380] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 23:52:43,432] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 23:52:43,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:43,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:44,188] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:44,189] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:52:44,321] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:52:44,322] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:52:44,352] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:44,378] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 23:52:44,406] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-10 23:52:44,494] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 23:52:45,051] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:52:50,057] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 23:52:50,143] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 23:52:50,149] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:50,149] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:50,905] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:50,906] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 23:52:51,045] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 23:52:51,047] - [remote_util:5249] INFO - b'ok'
[2022-09-10 23:52:51,047] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 23:52:51,664] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 23:52:51,721] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 23:52:51,721] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 23:52:52,607] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:52:52,915] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:52:53,372] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 23:52:53,374] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-10 23:52:53,427] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:53,427] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:54,115] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:54,120] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:54,120] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:55,211] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:55,218] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:55,218] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:56,489] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:52:56,497] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:52:56,497] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:52:57,805] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:53:05,166] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:53:05,166] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.16221894908591, 'mem_free': 12935106560, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:53:05,166] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:53:05,167] - [basetestcase:467] INFO - Time to execute basesetup : 26.647404193878174
[2022-09-10 23:53:05,217] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:53:05,217] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:53:05,269] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:53:05,269] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:53:05,322] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:53:05,322] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:53:05,373] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 23:53:05,374] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 23:53:05,423] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:53:05,484] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-10 23:53:05,484] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 23:53:05,484] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 23:53:10,497] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 23:53:10,500] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:53:10,500] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:53:11,223] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:53:12,312] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 23:53:12,481] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 23:53:14,623] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 23:53:14,707] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:53:14,707] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 23:53:14,707] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 23:53:44,733] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 23:53:44,763] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:53:44,789] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:53:44,858] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.639595ms
[2022-09-10 23:53:44,858] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '496b6223-34e6-4d8d-a356-ace42da7f608', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.639595ms', 'executionTime': '67.583233ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 23:53:44,859] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 23:53:44,885] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 23:53:44,910] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 23:53:45,639] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 726.537292ms
[2022-09-10 23:53:45,639] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 23:53:45,691] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:53:45,736] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:53:45,755] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 16.41419ms
[2022-09-10 23:53:45,991] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:53:46,029] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 23:53:46,045] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 23:53:46,046] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 23:53:46,065] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 23:53:46,137] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:53:46,283] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:53:46,286] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:53:46,286] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:53:47,081] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:53:47,141] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:53:47,141] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 23:53:47,273] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:53:47,335] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:53:47,338] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:53:47,338] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:53:48,136] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:53:48,195] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 23:53:48,195] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 23:53:48,324] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 23:53:48,379] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 23:53:48,379] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 23:53:48,379] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:53:48,404] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 23:53:48,429] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 23:53:48,439] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.993752ms
[2022-09-10 23:53:48,465] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 23:53:48,490] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 23:53:48,549] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.519245ms
[2022-09-10 23:53:48,611] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 23:53:48,611] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.23141081528581, 'mem_free': 12744531968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 23:53:48,611] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 23:53:48,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:53:48,616] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:53:49,408] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:53:49,413] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:53:49,413] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:53:50,170] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:53:50,175] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:53:50,176] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:53:51,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:53:51,388] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 23:53:51,388] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 23:53:52,729] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 23:54:01,608] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-10 23:54:01,764] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 23:54:02,297] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 23:54:02,325] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 23:54:02,326] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 23:54:02,382] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:54:02,439] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:54:02,496] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 23:54:02,497] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 23:54:02,582] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 23:54:02,583] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 23:54:02,610] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:54:02,750] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 23:54:02,750] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:54:02,780] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 23:54:02,809] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 23:54:02,809] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:54:02,836] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 23:54:02,863] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 23:54:02,863] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:54:02,905] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 23:54:02,933] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 23:54:02,933] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 23:54:02,961] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 23:54:02,962] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-10 23:54:02,962] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 23:54:02,963] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 84.529s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 25368 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-10.09.2022-22.04.cfg
Build Log: make-10.09.2022-22.04.log
Server Log: logs-10.09.2022-22.04.tar.gz

Finished