Building

Started building at 2022/09/10 18:02:24
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 10.09.2022-18.02
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at de1130b, changes since last good build: none
Switching plasma to unstable
plasma is at a338d0e, changes since last good build: 
fatal: Invalid revision range 70ba222be48a88c87394d7fa22445b17df4a5ec1..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable d5434ebf] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable b5cb5eee] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 4fe86675] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable 6948ddc4] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable e732a8fc] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 4a4306a0] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable 287055aa] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable 2bfa77b3] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable 0d1cb4ab] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 57105f60] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable fcb6b8ae] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable c29a8452] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179227 at refs/changes/27/179227/10
Total 8 (delta 3), reused 4 (delta 3)
[unstable eb2beb76] MB-51825: Remove vbuckets arg from indexer command line and its usage
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Aug 25 18:17:32 2022 +0530
 2 files changed, 9 deletions(-)
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 7b6164e6] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 261285a] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable ccc7934] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 1f2b774] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable c776a36] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 6a1dd76] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable 7044586] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable a036428] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 4d91499] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Pulling in uncommitted change 179769 at refs/changes/69/179769/2
Total 5 (delta 4), reused 5 (delta 4)
[unstable 547c303] MB-52096: Use memory stats to estimate mutation quota if page stats not available
 Author: jliang00 
 Date: Thu Sep 8 16:56:16 2022 -0700
 3 files changed, 86 insertions(+), 27 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/10 18:56:51
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Sep 10 19:02:14 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 19:06:07 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 19:06:51 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 10 19:08:19 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 10 19:09:11 b" 'stop_on_failure': 'True'}"
Sep 10 19:09:11 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 19:09:11 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_1'}"
Sep 10 19:09:11 b'-->result: '
Sep 10 19:09:11 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 10 19:09:11 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 19:09:11 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_2'}"
Sep 10 19:09:11 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 19:09:11 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 10 19:09:11 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_3'}"
Sep 10 19:09:11 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 19:09:11 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 19:09:11 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 10 19:09:11 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_4'}"
Sep 10 19:09:11 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 19:09:11 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 19:09:11 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 19:18:38 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 10 19:19:23 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 10 19:23:08 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 19:27:04 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 19:29:46 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 19:29:46 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_5'}"
Sep 10 19:29:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 19:29:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 10 19:29:46 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 10 19:29:46 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_6'}"
Sep 10 19:29:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 19:29:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 19:29:46 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 10 19:29:46 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_7'}"
Sep 10 19:29:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 19:29:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 19:29:46 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 10 19:29:46 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 10 19:29:46 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_18-57-12/test_8'}"
Sep 10 19:29:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 19:29:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 19:29:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 19:29:46 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 10 19:29:46 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 10 19:29:47 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 19:29:47 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 19:29:47 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 10 19:29:47 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 10 19:29:47 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 10 19:29:47 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 10 19:29:47 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 10 19:29:47 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 180: go: command not found
/home/buildbot/bin/dotest: line 183: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-10 19:32:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:32:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:32:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:32:39 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 19:32:42 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 19:32:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 19:33:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:08 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-10 19:33:09 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 2143707136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:33:09 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 19:33:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:33:10 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 1968062464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:33:10 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 19:33:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:33:11 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1833558016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:33:11 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:33:12 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1753063424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:33:12 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:33:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:33:13 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:33:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:33:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:33:13 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 19:33:13 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 19:33:13 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-10 19:33:13 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 19:33:18 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 19:33:18 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-10 19:33:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:33:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:33:19 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:33:19 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 19:33:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 19:33:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-10 19:33:19 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 19:33:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 19:33:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 19:33:21 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 49.33116542434593, 'mem_free': 13407694848, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.86365485191345
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 19:33:24 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 19:33:29 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 19:33:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:33:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:33:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:33:30 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 19:33:31 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 19:33:32 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 19:33:32 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:33:32 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 19:33:32 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 19:34:02 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 132.709372ms
2022-09-10 19:34:03 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '2ea2960f-5520-4541-bd33-744272f443c0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '132.709372ms', 'executionTime': '132.572242ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 699.615771ms
2022-09-10 19:34:03 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.783116ms
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.787702ms
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 49.088339ms
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.538925048276215, 'mem_free': 13367635968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:09 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 19:34:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 19:34:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 19:34:11 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 92.127s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-10 19:34:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.25965702849417, 'mem_free': 13347438592, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 19:34:17 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 19:34:17 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-10 19:34:18 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 19:34:18 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13347438592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 19:34:19 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13341777920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13342326784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13341384704, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:34:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:34:21 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:34:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 19:34:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-10 19:34:21 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 19:34:21 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 19:34:26 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 19:34:26 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-10 19:34:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 19:34:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 19:34:27 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 19:34:27 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 19:34:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 19:34:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-10 19:34:27 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 19:34:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 19:34:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 19:34:28 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 19:34:28 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.70515259064585, 'mem_free': 13332914176, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.39651322364807
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 19:34:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 19:34:38 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 19:34:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:34:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:34:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:34:39 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 19:34:39 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 19:34:42 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 19:34:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:34:42 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 19:34:42 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 19:35:12 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 19:35:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 19:35:12 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 19:35:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 73.133401ms
2022-09-10 19:35:12 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '084fe515-0cd3-46cb-b48c-9a59f322ebea', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '73.133401ms', 'executionTime': '73.060634ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 19:35:12 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 19:35:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 19:35:12 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 692.323226ms
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.73664ms
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 19:35:13 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 19:35:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:35:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee36b8407358344bf187ce510af907294ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-10 19:35:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee36b8407358344bf187ce510af907294ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-10 19:35:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 31.753861ms
2022-09-10 19:35:14 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee36b8407358344bf187ce510af907294ajob_title) USING GSI
2022-09-10 19:35:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee36b8407358344bf187ce510af907294ajob_title) USING GSI
2022-09-10 19:35:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee36b8407358344bf187ce510af907294ajob_title%29+USING+GSI
2022-09-10 19:35:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 23.159099ms
2022-09-10 19:35:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee36b8407358344bf187ce510af907294ajob_title'
2022-09-10 19:35:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee36b8407358344bf187ce510af907294ajob_title%27
2022-09-10 19:35:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.817785ms
2022-09-10 19:35:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee36b8407358344bf187ce510af907294ajob_title'
2022-09-10 19:35:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee36b8407358344bf187ce510af907294ajob_title%27
2022-09-10 19:35:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.436276ms
2022-09-10 19:35:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 19:35:19 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 19:35:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-10 19:35:19 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.530939ms
2022-09-10 19:35:19 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '12a4b466-1690-41bc-b7ba-dcc033d5e562', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee36b8407358344bf187ce510af907294ajob_title', 'index_id': '8e5f8961abb1626d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.530939ms', 'executionTime': '2.472828ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-10 19:35:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 19:35:19 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 19:35:19 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 19:35:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-10 19:35:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-10 19:35:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-10 19:35:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-10 19:35:19 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-10 19:35:19 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-10 19:35:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 19:35:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 19:35:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-10 19:35:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 95.573356ms
2022-09-10 19:35:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-10 19:35:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-10 19:35:21 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 19:35:21 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee36b8407358344bf187ce510af907294ajob_title'
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee36b8407358344bf187ce510af907294ajob_title%27
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.794629ms
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee36b8407358344bf187ce510af907294ajob_title ON default USING GSI
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee36b8407358344bf187ce510af907294ajob_title+ON+default+USING+GSI
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 45.810409ms
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee36b8407358344bf187ce510af907294ajob_title'
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee36b8407358344bf187ce510af907294ajob_title%27
2022-09-10 19:35:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.096122ms
2022-09-10 19:35:22 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '803a2db6-4ef4-4c68-b978-2f315c575cee', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.096122ms', 'executionTime': '4.972347ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.352287ms
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 40.525425ms
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.3998877763864, 'mem_free': 13215621120, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:35:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:35:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:35:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:35:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:35:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:35:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 19:35:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 19:35:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 19:35:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 19:35:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 19:35:28 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 19:35:28 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 19:35:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 19:35:29 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 19:35:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.808s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_2'}
[2022-09-10 19:35:29,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:29,632] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:29,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:29,940] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:35:30,022] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:35:30,023] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-10 19:35:30,023] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:35:30,663] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:35:30,691] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13261967360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:35:30,719] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:35:30,720] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:35:30,720] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:35:30,762] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:35:30,795] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:35:30,795] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:35:30,825] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:35:30,826] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:35:30,826] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:35:30,826] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:35:30,877] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:35:30,880] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:30,880] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:31,156] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:31,158] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:35:31,231] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:35:31,232] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:35:31,263] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:35:31,264] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:35:31,390] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:35:31,417] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13206265856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:35:31,444] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:35:31,471] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:35:31,471] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:35:31,522] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:35:31,525] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:31,525] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:31,814] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:31,815] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:35:31,893] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:35:31,894] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:35:31,927] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:35:31,927] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:35:32,052] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:35:32,079] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13206794240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:35:32,106] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:35:32,136] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:35:32,136] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:35:32,189] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:35:32,194] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:32,194] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:32,494] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:32,495] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:35:32,570] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:35:32,571] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:35:32,603] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:35:32,603] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:35:32,726] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:35:32,759] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13230710784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:35:32,787] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:35:32,815] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:35:32,815] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:35:32,868] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:35:32,871] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:32,872] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:33,165] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:33,166] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:35:33,240] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:35:33,241] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:35:33,273] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:35:33,273] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:35:33,370] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:35:34,053] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:35:39,056] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:35:39,139] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:35:39,142] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:39,142] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:39,416] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:39,417] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:35:39,486] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:35:39,487] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:35:39,488] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:35:40,484] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:35:40,544] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 19:35:40,545] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:35:41,532] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:35:41,855] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:35:42,152] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:35:42,155] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:35:42,208] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:42,209] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:42,460] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:42,465] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:42,465] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:42,713] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:42,718] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:42,718] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:42,973] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:42,978] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:42,978] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:43,326] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:47,076] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:35:47,076] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.26973444707144, 'mem_free': 13258260480, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:35:47,076] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:35:47,077] - [basetestcase:467] INFO - Time to execute basesetup : 17.44689440727234
[2022-09-10 19:35:47,128] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:35:47,129] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:35:47,184] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:35:47,184] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:35:47,237] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:35:47,237] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:35:47,290] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:35:47,291] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:35:47,340] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:35:47,399] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:35:47,399] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:35:47,400] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:35:52,415] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:35:52,418] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:35:52,418] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:35:52,673] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:35:53,603] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:35:53,772] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:35:56,465] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:35:56,488] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 19:35:57,148] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:35:57,148] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:35:57,148] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:36:27,169] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:36:27,199] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:36:27,224] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:36:27,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.976022ms
[2022-09-10 19:36:27,295] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'dd5363fd-1065-444b-bb7d-1fce4e6702f6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.976022ms', 'executionTime': '67.853623ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:36:27,295] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:36:27,322] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:36:27,347] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:36:28,040] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 690.924672ms
[2022-09-10 19:36:28,040] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:36:28,114] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:36:28,155] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:36:28,165] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.821896ms
[2022-09-10 19:36:28,370] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:36:28,406] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:36:28,417] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:36:28,418] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:36:28,435] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:36:28,508] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:36:29,331] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee054f2806b5444772a25f6a6f1d4320e4job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:36:29,357] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee054f2806b5444772a25f6a6f1d4320e4job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:36:29,387] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 27.619495ms
[2022-09-10 19:36:29,387] - [base_gsi:282] INFO - BUILD INDEX on default(employee054f2806b5444772a25f6a6f1d4320e4job_title) USING GSI
[2022-09-10 19:36:30,418] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee054f2806b5444772a25f6a6f1d4320e4job_title) USING GSI
[2022-09-10 19:36:30,445] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee054f2806b5444772a25f6a6f1d4320e4job_title%29+USING+GSI
[2022-09-10 19:36:30,476] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.726126ms
[2022-09-10 19:36:31,509] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee054f2806b5444772a25f6a6f1d4320e4job_title'
[2022-09-10 19:36:31,536] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee054f2806b5444772a25f6a6f1d4320e4job_title%27
[2022-09-10 19:36:31,543] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.834267ms
[2022-09-10 19:36:32,574] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee054f2806b5444772a25f6a6f1d4320e4job_title'
[2022-09-10 19:36:32,601] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee054f2806b5444772a25f6a6f1d4320e4job_title%27
[2022-09-10 19:36:32,609] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.937278ms
[2022-09-10 19:36:33,066] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 19:36:33,232] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:36:34,132] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:36:34,612] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:36:34,646] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:36:34,677] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 19:36:34,681] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.660923ms
[2022-09-10 19:36:34,681] - [task:3246] INFO - {'requestID': '02256f44-68b9-48d9-9b11-45134926ecb6', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee054f2806b5444772a25f6a6f1d4320e4job_title', 'index_id': '17b2c2026ed80b52', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.660923ms', 'executionTime': '2.573918ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 19:36:34,682] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:36:34,682] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:36:34,683] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:36:34,683] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:36:34,683] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 19:36:34,683] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:36:34,684] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:36:34,684] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:36:34,684] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:36:35,684] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:36:35,714] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:36:35,741] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 19:36:35,858] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 107.393653ms
[2022-09-10 19:36:35,859] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:36:35,859] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:36:36,648] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:36:36,648] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:36:37,681] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee054f2806b5444772a25f6a6f1d4320e4job_title'
[2022-09-10 19:36:37,708] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee054f2806b5444772a25f6a6f1d4320e4job_title%27
[2022-09-10 19:36:37,720] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.722183ms
[2022-09-10 19:36:37,746] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee054f2806b5444772a25f6a6f1d4320e4job_title ON default USING GSI
[2022-09-10 19:36:37,772] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee054f2806b5444772a25f6a6f1d4320e4job_title+ON+default+USING+GSI
[2022-09-10 19:36:37,828] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.806175ms
[2022-09-10 19:36:37,863] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee054f2806b5444772a25f6a6f1d4320e4job_title'
[2022-09-10 19:36:37,889] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee054f2806b5444772a25f6a6f1d4320e4job_title%27
[2022-09-10 19:36:37,897] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.150901ms
[2022-09-10 19:36:37,898] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c75f7883-73ed-49b5-8068-ba59dba57f8a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.150901ms', 'executionTime': '6.092573ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:36:38,002] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:36:38,005] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:38,005] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:38,353] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:38,409] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:36:38,409] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:36:38,469] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:36:38,524] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:36:38,527] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:38,527] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:38,894] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:38,952] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:36:38,952] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:36:39,016] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:36:39,073] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:36:39,073] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:36:39,074] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:36:39,102] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:36:39,132] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:36:39,141] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.835291ms
[2022-09-10 19:36:39,170] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:36:39,201] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:36:39,240] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.050811ms
[2022-09-10 19:36:39,320] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:36:39,320] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.473764101566503, 'mem_free': 13158703104, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:36:39,321] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:36:39,325] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:39,326] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:39,703] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:39,708] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:39,708] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:40,071] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:40,076] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:40,076] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:40,617] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:40,625] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:40,625] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:41,247] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:45,280] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:36:45,424] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:36:46,071] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:36:46,100] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:36:46,100] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:36:46,154] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:36:46,208] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:36:46,262] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:36:46,263] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:36:46,344] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:36:46,345] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:36:46,371] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:36:46,502] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:36:46,503] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:36:46,530] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:36:46,572] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:36:46,573] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:36:46,601] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:36:46,627] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:36:46,627] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:36:46,654] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:36:46,680] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:36:46,681] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:36:46,707] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:36:46,708] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:36:46,708] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:36:46,708] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 77.130s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_3'}
[2022-09-10 19:36:46,806] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:46,806] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:47,147] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:47,178] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:36:47,258] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:36:47,259] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-10 19:36:47,259] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:36:47,836] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:36:47,864] - [task:165] INFO -  {'uptime': '244', 'memoryTotal': 15466930176, 'memoryFree': 13128151040, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:36:47,891] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:36:47,892] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:36:47,892] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:36:47,931] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:36:47,964] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:36:47,965] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:36:47,994] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:36:47,995] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:36:47,995] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:36:47,995] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:36:48,045] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:36:48,048] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:48,048] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:48,377] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:48,378] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:36:48,458] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:36:48,460] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:36:48,492] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:36:48,493] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:36:48,622] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:36:48,651] - [task:165] INFO -  {'uptime': '240', 'memoryTotal': 15466930176, 'memoryFree': 13190012928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:36:48,677] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:36:48,705] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:36:48,706] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:36:48,759] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:36:48,762] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:48,762] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:49,100] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:49,101] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:36:49,192] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:36:49,193] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:36:49,225] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:36:49,226] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:36:49,350] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:36:49,377] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13190778880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:36:49,405] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:36:49,434] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:36:49,434] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:36:49,488] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:36:49,493] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:49,493] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:49,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:49,810] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:36:49,887] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:36:49,888] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:36:49,919] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:36:49,920] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:36:50,036] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:36:50,062] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13191045120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:36:50,088] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:36:50,116] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:36:50,116] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:36:50,168] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:36:50,171] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:50,171] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:50,497] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:50,498] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:36:50,575] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:36:50,576] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:36:50,607] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:36:50,608] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:36:50,703] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:36:51,302] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:36:56,306] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:36:56,390] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:36:56,395] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:56,395] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:56,737] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:56,738] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:36:56,815] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:36:56,817] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:36:56,817] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:36:57,841] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:36:57,897] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 19:36:57,898] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:36:58,827] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:36:59,154] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:36:59,433] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:36:59,436] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:36:59,494] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:59,494] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:36:59,808] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:36:59,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:36:59,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:00,138] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:00,143] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:00,143] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:00,501] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:00,509] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:00,509] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:01,093] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:05,242] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:37:05,243] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.52001282843488, 'mem_free': 13195407360, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:37:05,243] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:37:05,243] - [basetestcase:467] INFO - Time to execute basesetup : 18.439261436462402
[2022-09-10 19:37:05,297] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:37:05,298] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:37:05,354] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:37:05,354] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:37:05,411] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:37:05,411] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:37:05,466] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:37:05,466] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:37:05,521] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:37:05,586] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:37:05,586] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:37:05,587] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:37:10,599] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:37:10,603] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:10,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:10,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:11,854] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:37:12,027] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:37:14,655] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:37:14,673] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 19:37:15,512] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:37:15,512] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:37:15,512] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:37:45,543] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:37:45,572] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:37:45,597] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:37:45,663] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.184847ms
[2022-09-10 19:37:45,664] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '400f4f2a-0b93-4c8a-b9a6-15dcf17dd603', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.184847ms', 'executionTime': '65.117601ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:37:45,664] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:37:45,689] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:37:45,713] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:37:46,412] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 696.913369ms
[2022-09-10 19:37:46,412] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:37:46,486] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:37:46,530] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:37:46,538] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.174956ms
[2022-09-10 19:37:46,724] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:37:46,759] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:37:46,773] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:37:46,775] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:37:46,799] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:37:46,870] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:37:47,692] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee06059b3481df40e3acd7adc8a63f5c95job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:37:47,717] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee06059b3481df40e3acd7adc8a63f5c95job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:37:47,754] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.394952ms
[2022-09-10 19:37:47,754] - [base_gsi:282] INFO - BUILD INDEX on default(employee06059b3481df40e3acd7adc8a63f5c95job_title) USING GSI
[2022-09-10 19:37:48,786] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee06059b3481df40e3acd7adc8a63f5c95job_title) USING GSI
[2022-09-10 19:37:48,813] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee06059b3481df40e3acd7adc8a63f5c95job_title%29+USING+GSI
[2022-09-10 19:37:48,837] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.046527ms
[2022-09-10 19:37:49,867] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06059b3481df40e3acd7adc8a63f5c95job_title'
[2022-09-10 19:37:49,894] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06059b3481df40e3acd7adc8a63f5c95job_title%27
[2022-09-10 19:37:49,903] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.708102ms
[2022-09-10 19:37:50,934] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06059b3481df40e3acd7adc8a63f5c95job_title'
[2022-09-10 19:37:50,959] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06059b3481df40e3acd7adc8a63f5c95job_title%27
[2022-09-10 19:37:50,969] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.765702ms
[2022-09-10 19:37:51,357] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 19:37:51,606] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:37:52,297] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:37:52,971] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:37:53,008] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:37:53,037] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 19:37:53,041] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.825329ms
[2022-09-10 19:37:53,041] - [task:3246] INFO - {'requestID': '8a549051-0d36-4fcf-a712-dc5837e79b90', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee06059b3481df40e3acd7adc8a63f5c95job_title', 'index_id': '8c08e5df443a9cd4', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.825329ms', 'executionTime': '2.75955ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 19:37:53,042] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:37:53,042] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:37:53,042] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:37:53,043] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:37:53,043] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 19:37:53,043] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:37:53,044] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:37:53,044] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:37:53,044] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:37:54,043] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:37:54,074] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:37:54,099] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 19:37:54,243] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 133.715253ms
[2022-09-10 19:37:54,243] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:37:54,244] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:37:55,026] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:37:55,026] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:37:56,057] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06059b3481df40e3acd7adc8a63f5c95job_title'
[2022-09-10 19:37:56,084] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06059b3481df40e3acd7adc8a63f5c95job_title%27
[2022-09-10 19:37:56,093] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.427314ms
[2022-09-10 19:37:56,119] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee06059b3481df40e3acd7adc8a63f5c95job_title ON default USING GSI
[2022-09-10 19:37:56,147] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee06059b3481df40e3acd7adc8a63f5c95job_title+ON+default+USING+GSI
[2022-09-10 19:37:56,189] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.623638ms
[2022-09-10 19:37:56,227] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06059b3481df40e3acd7adc8a63f5c95job_title'
[2022-09-10 19:37:56,255] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06059b3481df40e3acd7adc8a63f5c95job_title%27
[2022-09-10 19:37:56,263] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.224567ms
[2022-09-10 19:37:56,264] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7ed3697c-87ec-4f3f-b43b-faf08906fd63', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.224567ms', 'executionTime': '6.098212ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:37:56,374] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:37:56,378] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:56,378] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:56,804] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:56,863] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:37:56,863] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:37:56,935] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:37:56,995] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:37:56,998] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:56,998] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:57,439] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:57,500] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:37:57,501] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:37:57,573] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:37:57,629] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:37:57,630] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:37:57,630] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:37:57,655] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:37:57,682] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:37:57,692] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.049462ms
[2022-09-10 19:37:57,718] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:37:57,742] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:37:57,796] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.679456ms
[2022-09-10 19:37:57,867] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:37:57,868] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.69829594740736, 'mem_free': 13074550784, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:37:57,868] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:37:57,872] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:57,872] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:58,268] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:58,273] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:58,273] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:58,934] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:58,942] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:58,942] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:37:59,625] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:37:59,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:37:59,633] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:00,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:04,654] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:38:04,806] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:38:05,327] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:38:05,355] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:38:05,355] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:38:05,412] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:38:05,469] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:38:05,525] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:38:05,526] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:38:05,606] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:38:05,607] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:38:05,632] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:38:05,760] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:38:05,760] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:38:05,786] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:38:05,812] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:38:05,812] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:38:05,839] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:38:05,864] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:38:05,864] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:38:05,892] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:38:05,919] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:38:05,919] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:38:05,963] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:38:05,963] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:38:05,963] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:38:05,964] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.213s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_4'}
[2022-09-10 19:38:06,065] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:06,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:06,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:06,459] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:38:06,538] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:38:06,538] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-10 19:38:06,539] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:38:07,096] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:38:07,126] - [task:165] INFO -  {'uptime': '323', 'memoryTotal': 15466930176, 'memoryFree': 13075697664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:38:07,159] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:38:07,159] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:38:07,160] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:38:07,196] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:38:07,232] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:38:07,232] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:38:07,261] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:38:07,261] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:38:07,262] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:38:07,262] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:38:07,313] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:38:07,316] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:07,316] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:07,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:07,721] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:38:07,812] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:38:07,813] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:38:07,845] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:38:07,846] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:38:07,976] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:38:08,006] - [task:165] INFO -  {'uptime': '320', 'memoryTotal': 15466930176, 'memoryFree': 13151936512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:38:08,034] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:38:08,064] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:38:08,065] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:38:08,118] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:38:08,121] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:08,122] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:08,513] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:08,514] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:38:08,596] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:38:08,597] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:38:08,627] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:38:08,627] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:38:08,745] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:38:08,772] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13152563200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:38:08,798] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:38:08,827] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:38:08,827] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:38:08,880] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:38:08,884] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:08,884] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:09,248] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:09,249] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:38:09,336] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:38:09,337] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:38:09,368] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:38:09,368] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:38:09,493] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:38:09,521] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13152489472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:38:09,548] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:38:09,580] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:38:09,585] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:38:09,636] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:38:09,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:09,639] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:10,035] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:10,037] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:38:10,121] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:38:10,122] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:38:10,152] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:38:10,153] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:38:10,247] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:38:10,785] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:38:15,790] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:38:15,874] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:38:15,880] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:15,880] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:16,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:16,284] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:38:16,365] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:38:16,366] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:38:16,366] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:38:17,385] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:38:17,440] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 19:38:17,440] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:38:18,361] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:38:18,691] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:38:18,979] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:38:18,982] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:38:19,041] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:19,041] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:19,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:19,424] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:19,425] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:19,801] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:19,806] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:19,806] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:20,340] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:20,349] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:20,349] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:20,993] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:25,518] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:38:25,518] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.69185737062264, 'mem_free': 13155180544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:38:25,518] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:38:25,519] - [basetestcase:467] INFO - Time to execute basesetup : 19.455636739730835
[2022-09-10 19:38:25,574] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:38:25,574] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:38:25,631] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:38:25,631] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:38:25,690] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:38:25,690] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:38:25,747] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:38:25,747] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:38:25,800] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:38:25,864] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:38:25,865] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:38:25,865] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:38:30,876] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:38:30,879] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:38:30,879] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:38:31,238] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:38:32,206] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:38:32,458] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:38:34,970] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:38:34,986] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 19:38:35,656] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:38:35,656] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:38:35,656] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:39:05,685] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:39:05,715] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:39:05,742] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:39:05,813] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.017911ms
[2022-09-10 19:39:05,813] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '22dd8100-89f1-45d4-8013-5c38da13a807', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.017911ms', 'executionTime': '68.960239ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:39:05,813] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:39:05,840] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:39:05,866] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:39:06,514] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 645.867186ms
[2022-09-10 19:39:06,515] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:39:06,577] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:39:06,613] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:39:06,621] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.821682ms
[2022-09-10 19:39:06,821] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:39:06,855] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:39:06,870] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:39:06,870] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:39:06,880] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:39:06,945] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:39:07,787] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:39:07,811] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:39:07,855] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.308066ms
[2022-09-10 19:39:07,856] - [base_gsi:282] INFO - BUILD INDEX on default(employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title) USING GSI
[2022-09-10 19:39:08,890] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title) USING GSI
[2022-09-10 19:39:08,917] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title%29+USING+GSI
[2022-09-10 19:39:08,942] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.938357ms
[2022-09-10 19:39:09,974] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title'
[2022-09-10 19:39:09,999] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title%27
[2022-09-10 19:39:10,008] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.93137ms
[2022-09-10 19:39:11,038] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title'
[2022-09-10 19:39:11,065] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title%27
[2022-09-10 19:39:11,075] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.623479ms
[2022-09-10 19:39:11,554] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 19:39:11,721] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:39:12,617] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:39:12,815] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-10 19:39:12,815] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:39:12,998] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-10 19:39:12,999] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-10 19:39:12,999] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (1784692196, 0, b'')
[2022-09-10 19:39:13,078] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:39:13,106] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:39:13,133] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 19:39:13,138] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.148564ms
[2022-09-10 19:39:13,138] - [task:3246] INFO - {'requestID': '70d933f6-0eeb-4684-b02d-effa13f32eca', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title', 'index_id': '584a3aab0aff2745', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.148564ms', 'executionTime': '3.084435ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 19:39:13,138] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:39:13,139] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:39:13,139] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:39:13,139] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:39:13,140] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 19:39:13,140] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:39:13,140] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:39:13,140] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:39:13,141] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:39:14,140] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:39:14,171] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:39:14,199] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 19:39:14,338] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 131.327087ms
[2022-09-10 19:39:14,339] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:39:14,339] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:39:15,130] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:39:15,131] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:39:16,162] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title'
[2022-09-10 19:39:16,191] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title%27
[2022-09-10 19:39:16,200] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.635728ms
[2022-09-10 19:39:16,227] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title ON default USING GSI
[2022-09-10 19:39:16,254] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title+ON+default+USING+GSI
[2022-09-10 19:39:16,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.018739ms
[2022-09-10 19:39:16,331] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title'
[2022-09-10 19:39:16,359] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9996ae0b603d4ce1a1caf30ebf1d9e43job_title%27
[2022-09-10 19:39:16,366] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.751692ms
[2022-09-10 19:39:16,367] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'dfc7cb22-02a3-4f21-ac7a-59505bdf10b8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.751692ms', 'executionTime': '5.671516ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:39:16,473] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:39:16,477] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:16,477] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:16,918] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:16,975] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:39:16,976] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:39:17,071] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:39:17,132] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:39:17,135] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:17,135] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:17,568] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:17,624] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:39:17,624] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:39:17,705] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:39:17,764] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:39:17,764] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:39:17,765] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:39:17,791] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:39:17,816] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:39:17,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.573403ms
[2022-09-10 19:39:17,852] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:39:17,877] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:39:17,930] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.651113ms
[2022-09-10 19:39:17,996] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:39:17,997] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.96691631803975, 'mem_free': 13017853952, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:39:17,997] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:39:18,001] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:18,001] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:18,430] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:18,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:18,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:18,984] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:18,996] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:18,996] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:19,741] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:19,748] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:19,749] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:20,535] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:25,535] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:39:25,677] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:39:26,044] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:39:26,073] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:39:26,073] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:39:26,129] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:39:26,186] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:39:26,242] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:39:26,243] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:39:26,325] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:39:26,326] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:39:26,352] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:39:26,485] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:39:26,485] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:39:26,512] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:39:26,539] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:39:26,539] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:39:26,566] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:39:26,592] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:39:26,592] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:39:26,619] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:39:26,646] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:39:26,646] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:39:26,675] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:39:26,675] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:39:26,675] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:39:26,676] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.669s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_5'}
[2022-09-10 19:39:26,782] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:26,783] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:27,250] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:27,285] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:39:27,368] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:39:27,369] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-10 19:39:27,369] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:39:27,808] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:39:27,834] - [task:165] INFO -  {'uptime': '404', 'memoryTotal': 15466930176, 'memoryFree': 13007327232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:39:27,860] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:39:27,860] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:39:27,861] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:39:27,897] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:39:27,931] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:39:27,932] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:39:27,960] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:39:27,961] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:39:27,961] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:39:27,962] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:39:28,010] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:39:28,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:28,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:28,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:28,418] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:39:28,505] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:39:28,505] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:39:28,536] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:39:28,536] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:39:28,657] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:39:28,685] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13120491520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:39:28,711] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:39:28,739] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:39:28,739] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:39:28,789] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:39:28,792] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:28,792] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:29,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:29,195] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:39:29,281] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:39:29,282] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:39:29,312] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:39:29,313] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:39:29,426] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:39:29,453] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13119877120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:39:29,478] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:39:29,506] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:39:29,506] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:39:29,556] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:39:29,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:29,560] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:29,999] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:30,000] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:39:30,093] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:39:30,094] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:39:30,124] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:39:30,124] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:39:30,243] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:39:30,270] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13119803392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:39:30,298] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:39:30,325] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:39:30,325] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:39:30,377] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:39:30,379] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:30,380] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:30,831] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:30,832] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:39:30,928] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:39:30,930] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:39:30,962] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:39:30,963] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:39:31,060] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:39:31,588] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:39:36,593] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:39:36,675] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:39:36,678] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:36,679] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:37,110] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:37,111] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:39:37,202] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:39:37,203] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:39:37,204] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:39:38,205] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:39:38,262] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 19:39:38,263] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:39:39,167] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:39:39,490] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:39:39,792] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:39:39,796] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:39:39,867] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:39,867] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:40,356] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:40,361] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:40,361] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:40,869] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:40,880] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:40,880] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:41,659] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:41,670] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:41,670] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:42,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:47,234] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:39:47,234] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.15390026809437, 'mem_free': 13125758976, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:39:47,235] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:39:47,235] - [basetestcase:467] INFO - Time to execute basesetup : 20.4562828540802
[2022-09-10 19:39:47,286] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:39:47,286] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:39:47,339] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:39:47,339] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:39:47,393] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:39:47,393] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:39:47,448] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:39:47,449] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:39:47,499] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:39:47,557] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:39:47,558] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:39:47,559] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:39:52,566] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:39:52,570] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:39:52,570] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:39:53,025] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:39:54,020] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:39:54,268] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:39:56,709] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:39:56,725] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 19:39:57,795] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:39:57,796] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:39:57,796] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:40:27,820] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:40:27,849] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:40:27,875] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:40:27,946] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.352748ms
[2022-09-10 19:40:27,947] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4cd253c1-6417-4c2c-9444-bcf9f003395b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.352748ms', 'executionTime': '69.152094ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:40:27,947] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:40:27,974] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:40:28,003] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:40:28,674] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 666.674881ms
[2022-09-10 19:40:28,674] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:40:28,731] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:40:28,769] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:40:28,776] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.429489ms
[2022-09-10 19:40:28,957] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:40:28,995] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:40:29,013] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:40:29,014] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:40:29,023] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:40:29,109] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:40:29,927] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee29572530f00c46319ce9730a4908475bjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:40:29,953] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee29572530f00c46319ce9730a4908475bjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:40:29,983] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.201968ms
[2022-09-10 19:40:29,984] - [base_gsi:282] INFO - BUILD INDEX on default(employee29572530f00c46319ce9730a4908475bjob_title) USING GSI
[2022-09-10 19:40:31,017] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee29572530f00c46319ce9730a4908475bjob_title) USING GSI
[2022-09-10 19:40:31,044] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee29572530f00c46319ce9730a4908475bjob_title%29+USING+GSI
[2022-09-10 19:40:31,066] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.772246ms
[2022-09-10 19:40:32,098] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee29572530f00c46319ce9730a4908475bjob_title'
[2022-09-10 19:40:32,124] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee29572530f00c46319ce9730a4908475bjob_title%27
[2022-09-10 19:40:32,131] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.478114ms
[2022-09-10 19:40:33,162] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee29572530f00c46319ce9730a4908475bjob_title'
[2022-09-10 19:40:33,190] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee29572530f00c46319ce9730a4908475bjob_title%27
[2022-09-10 19:40:33,197] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.232509ms
[2022-09-10 19:40:33,677] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:40:33,850] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:40:36,573] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:40:37,201] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:40:37,234] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:40:37,261] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 19:40:37,266] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.889209ms
[2022-09-10 19:40:37,266] - [task:3246] INFO - {'requestID': '53e9381e-3bba-4272-bd03-5282a13f7efb', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee29572530f00c46319ce9730a4908475bjob_title', 'index_id': '36f9217c6834b628', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.889209ms', 'executionTime': '2.819331ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 19:40:37,266] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:40:37,266] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:40:37,267] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:40:37,267] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:40:37,267] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 19:40:37,268] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:40:37,268] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:40:37,268] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:40:37,268] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:40:38,268] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:40:38,301] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:40:38,329] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 19:40:38,525] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 181.348928ms
[2022-09-10 19:40:38,526] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:40:38,527] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:40:40,095] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:40:40,095] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:40:41,127] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee29572530f00c46319ce9730a4908475bjob_title'
[2022-09-10 19:40:41,153] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee29572530f00c46319ce9730a4908475bjob_title%27
[2022-09-10 19:40:41,162] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.872282ms
[2022-09-10 19:40:41,189] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee29572530f00c46319ce9730a4908475bjob_title ON default USING GSI
[2022-09-10 19:40:41,215] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee29572530f00c46319ce9730a4908475bjob_title+ON+default+USING+GSI
[2022-09-10 19:40:41,262] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.929529ms
[2022-09-10 19:40:41,300] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee29572530f00c46319ce9730a4908475bjob_title'
[2022-09-10 19:40:41,325] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee29572530f00c46319ce9730a4908475bjob_title%27
[2022-09-10 19:40:41,332] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.369301ms
[2022-09-10 19:40:41,332] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd8dbb562-fdf5-4c70-bc48-c555fa511053', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.369301ms', 'executionTime': '5.297707ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:40:41,434] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:40:41,438] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:41,438] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:41,980] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:42,039] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:40:42,039] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:40:42,128] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:40:42,187] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:40:42,190] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:42,190] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:42,760] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:42,819] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:40:42,819] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:40:42,911] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:40:42,969] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:40:42,970] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:40:42,970] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:40:42,997] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:40:43,027] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:40:43,038] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.4749ms
[2022-09-10 19:40:43,071] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:40:43,099] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:40:43,166] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.389767ms
[2022-09-10 19:40:43,229] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:40:43,229] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 37.38829207550989, 'mem_free': 12964151296, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:40:43,229] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:40:43,233] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:43,233] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:43,786] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:43,791] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:43,791] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:44,325] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:44,329] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:44,329] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:45,147] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:45,155] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:45,155] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:46,085] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:52,079] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:40:52,219] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:40:52,766] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:40:52,795] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:40:52,795] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:40:52,852] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:40:52,907] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:40:52,958] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:40:52,959] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:40:53,043] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:40:53,044] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:40:53,070] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:40:53,214] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:40:53,214] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:40:53,241] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:40:53,267] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:40:53,267] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:40:53,295] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:40:53,320] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:40:53,321] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:40:53,356] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:40:53,383] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:40:53,384] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:40:53,411] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:40:53,411] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:40:53,411] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:40:53,412] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 86.686s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_6'}
[2022-09-10 19:40:53,517] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:53,517] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:54,074] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:54,107] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:40:54,192] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:40:54,193] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-10 19:40:54,193] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:40:54,544] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:40:54,571] - [task:165] INFO -  {'uptime': '491', 'memoryTotal': 15466930176, 'memoryFree': 13063671808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:40:54,598] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:40:54,598] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:40:54,598] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:40:54,634] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:40:54,669] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:40:54,670] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:40:54,698] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:40:54,699] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:40:54,699] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:40:54,699] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:40:54,748] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:40:54,750] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:54,751] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:55,284] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:55,285] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:40:55,396] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:40:55,397] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:40:55,426] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:40:55,426] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:40:55,556] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:40:55,584] - [task:165] INFO -  {'uptime': '491', 'memoryTotal': 15466930176, 'memoryFree': 13057081344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:40:55,612] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:40:55,642] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:40:55,643] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:40:55,695] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:40:55,698] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:55,698] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:56,185] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:56,187] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:40:56,292] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:40:56,293] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:40:56,323] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:40:56,324] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:40:56,441] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:40:56,468] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13065093120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:40:56,494] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:40:56,522] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:40:56,523] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:40:56,574] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:40:56,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:56,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:57,122] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:57,123] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:40:57,236] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:40:57,237] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:40:57,269] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:40:57,270] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:40:57,400] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:40:57,429] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13068656640, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:40:57,458] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:40:57,488] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:40:57,488] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:40:57,542] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:40:57,545] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:40:57,545] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:40:58,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:40:58,107] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:40:58,221] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:40:58,222] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:40:58,252] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:40:58,253] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:40:58,348] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:40:59,014] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:41:04,020] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:41:04,108] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:41:04,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:41:04,113] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:41:04,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:41:04,684] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:41:04,797] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:41:04,798] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:41:04,798] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:41:05,469] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:41:05,533] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 19:41:05,534] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:41:06,469] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:41:06,776] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:41:07,072] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:41:07,075] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:41:07,154] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:41:07,154] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:41:07,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:41:07,698] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:41:07,698] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:41:08,418] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:41:08,430] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:41:08,430] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:41:09,363] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:41:09,371] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:41:09,371] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:41:10,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:41:15,858] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:41:15,859] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.6849643966551, 'mem_free': 13058613248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:41:15,859] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:41:15,859] - [basetestcase:467] INFO - Time to execute basesetup : 22.344139575958252
[2022-09-10 19:41:15,913] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:41:15,914] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:41:15,969] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:41:15,969] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:41:16,024] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:41:16,024] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:41:16,080] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:41:16,081] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:41:16,132] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:41:16,197] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:41:16,198] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:41:16,198] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:41:21,208] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:41:21,211] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:41:21,212] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:41:21,733] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:41:22,739] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:41:23,009] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:41:25,344] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:41:25,369] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 19:41:26,747] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:41:26,748] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:41:26,748] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:41:56,756] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:41:56,786] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:41:56,812] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:41:56,884] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.318747ms
[2022-09-10 19:41:56,884] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8fdbc855-2b16-4855-80b0-64b9f8539425', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.318747ms', 'executionTime': '69.22824ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:41:56,884] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:41:56,911] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:41:56,938] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:41:57,620] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 680.131786ms
[2022-09-10 19:41:57,620] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:41:57,706] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:41:57,738] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:41:57,746] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.166209ms
[2022-09-10 19:41:57,946] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:41:57,986] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:41:58,006] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:41:58,007] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:41:58,020] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:41:58,087] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:41:58,900] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:41:58,927] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:41:58,978] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.776528ms
[2022-09-10 19:41:58,978] - [base_gsi:282] INFO - BUILD INDEX on default(employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title) USING GSI
[2022-09-10 19:42:00,013] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title) USING GSI
[2022-09-10 19:42:00,041] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title%29+USING+GSI
[2022-09-10 19:42:00,066] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.538408ms
[2022-09-10 19:42:01,096] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title'
[2022-09-10 19:42:01,122] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title%27
[2022-09-10 19:42:01,132] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.897195ms
[2022-09-10 19:42:02,164] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title'
[2022-09-10 19:42:02,193] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title%27
[2022-09-10 19:42:02,200] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.848922ms
[2022-09-10 19:42:02,706] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 19:42:02,883] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:42:04,008] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:42:04,620] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 19:42:04,794] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:42:06,091] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:42:06,501] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:42:06,676] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:42:10,330] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:42:11,221] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:42:11,254] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:42:11,280] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 19:42:11,284] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.980439ms
[2022-09-10 19:42:11,284] - [task:3246] INFO - {'requestID': '6a0c995f-3577-4418-98e3-025ec7ed0688', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title', 'index_id': 'c260b80b1884b81b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.980439ms', 'executionTime': '2.912804ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 19:42:11,285] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:42:11,285] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:42:11,285] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:42:11,286] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:42:11,286] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 19:42:11,286] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:42:11,287] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:42:11,287] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:42:11,287] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 19:42:12,287] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:42:12,317] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 19:42:12,343] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 19:42:12,754] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 187.319532ms
[2022-09-10 19:42:12,754] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:42:12,755] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:42:14,324] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:42:14,324] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 19:42:15,355] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title'
[2022-09-10 19:42:15,380] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title%27
[2022-09-10 19:42:15,390] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.141512ms
[2022-09-10 19:42:15,416] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title ON default USING GSI
[2022-09-10 19:42:15,442] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title+ON+default+USING+GSI
[2022-09-10 19:42:15,485] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.737121ms
[2022-09-10 19:42:15,520] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title'
[2022-09-10 19:42:15,546] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee59e0dbca1d9045b7997a2f76a0c9abbcjob_title%27
[2022-09-10 19:42:15,556] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.263925ms
[2022-09-10 19:42:15,557] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '95fb96fb-666d-46c4-a932-35ec1a42580e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.263925ms', 'executionTime': '8.187649ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:42:15,665] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:42:15,668] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:15,668] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:16,301] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:16,361] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:42:16,361] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:42:16,468] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:42:16,528] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:42:16,531] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:16,531] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:17,226] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:17,285] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:42:17,286] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:42:17,398] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:42:17,459] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:42:17,460] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:42:17,460] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:42:17,488] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:42:17,515] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:42:17,526] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.452939ms
[2022-09-10 19:42:17,554] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:42:17,579] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:42:17,638] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.365804ms
[2022-09-10 19:42:17,701] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:42:17,701] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.80065918597342, 'mem_free': 12840800256, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:42:17,702] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:42:17,706] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:17,706] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:18,389] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:18,394] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:18,394] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:19,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:19,090] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:19,091] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:20,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:20,084] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:20,085] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:21,268] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:28,756] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:42:28,902] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:42:29,993] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:42:30,034] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:42:30,035] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:42:30,091] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:42:30,147] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:42:30,204] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:42:30,205] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:42:30,285] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:42:30,287] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:42:30,313] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:42:30,441] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:42:30,442] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:42:30,468] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:42:30,493] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:42:30,493] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:42:30,520] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:42:30,545] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:42:30,545] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:42:30,573] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:42:30,599] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:42:30,599] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:42:30,626] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:42:30,626] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 19:42:30,626] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:42:30,627] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 97.167s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_7'}
[2022-09-10 19:42:30,769] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:30,770] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:31,350] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:31,384] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:42:31,462] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:42:31,463] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-10 19:42:31,463] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:42:31,797] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:42:31,826] - [task:165] INFO -  {'uptime': '588', 'memoryTotal': 15466930176, 'memoryFree': 13002530816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:42:31,853] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:42:31,853] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:42:31,854] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:42:31,889] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:42:31,929] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:42:31,929] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:42:31,959] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:42:31,960] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:42:31,961] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:42:31,961] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:42:32,011] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:42:32,014] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:32,014] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:32,648] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:32,649] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:42:32,778] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:42:32,779] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:42:32,809] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:42:32,810] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:42:32,938] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:42:32,965] - [task:165] INFO -  {'uptime': '586', 'memoryTotal': 15466930176, 'memoryFree': 13003411456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:42:32,991] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:42:33,022] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:42:33,023] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:42:33,074] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:42:33,076] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:33,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:33,664] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:33,665] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:42:33,777] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:42:33,778] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:42:33,809] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:42:33,810] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:42:33,930] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:42:33,964] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 13019295744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:42:33,991] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:42:34,024] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:42:34,024] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:42:34,076] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:42:34,080] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:34,080] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:34,725] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:34,727] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:42:34,850] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:42:34,851] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:42:34,883] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:42:34,884] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:42:35,013] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:42:35,044] - [task:165] INFO -  {'uptime': '583', 'memoryTotal': 15466930176, 'memoryFree': 13003907072, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:42:35,072] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:42:35,102] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:42:35,102] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:42:35,156] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:42:35,159] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:35,159] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:35,811] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:35,813] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:42:35,938] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:42:35,940] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:42:35,972] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:42:35,972] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:42:36,069] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:42:36,619] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:42:41,625] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:42:41,709] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:42:41,714] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:41,714] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:42,312] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:42,313] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:42:42,436] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:42:42,437] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:42:42,438] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:42:43,228] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:42:43,285] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 19:42:43,287] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:42:44,180] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:42:44,480] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:42:44,755] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:42:44,758] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 19:42:44,815] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:44,815] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:45,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:45,434] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:45,434] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:46,298] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:46,307] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:46,307] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:47,371] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:47,382] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:42:47,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:42:48,445] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:42:55,087] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:42:55,088] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.56604528743132, 'mem_free': 13017452544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:42:55,088] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:42:55,088] - [basetestcase:467] INFO - Time to execute basesetup : 24.321343183517456
[2022-09-10 19:42:55,138] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:42:55,139] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:42:55,194] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:42:55,194] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:42:55,248] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:42:55,249] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:42:55,305] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:42:55,305] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:42:55,359] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:42:55,426] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:42:55,426] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:42:55,427] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:43:00,440] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:43:00,444] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:00,445] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:01,111] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:02,168] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:43:02,345] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:43:04,555] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:43:04,637] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:43:04,637] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:43:04,637] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:43:34,655] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:43:34,685] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:43:34,711] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:43:34,781] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.595132ms
[2022-09-10 19:43:34,781] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9bdc70ff-3638-4c28-a105-a204dff7f7a0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.595132ms', 'executionTime': '67.533218ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:43:34,782] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:43:34,808] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:43:34,833] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:43:35,485] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 649.96122ms
[2022-09-10 19:43:35,485] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:43:35,553] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:43:35,592] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:43:35,600] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.54274ms
[2022-09-10 19:43:35,781] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:43:35,816] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:43:35,836] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:43:35,836] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:43:35,849] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:43:35,851] - [base_gsi:326] INFO - []
[2022-09-10 19:43:36,753] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee8998a9981b8442a3b867b941a94d183cjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:43:36,779] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee8998a9981b8442a3b867b941a94d183cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:43:36,813] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.134209ms
[2022-09-10 19:43:36,846] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee8998a9981b8442a3b867b941a94d183cjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:43:36,872] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee8998a9981b8442a3b867b941a94d183cjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:43:36,908] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.698216ms
[2022-09-10 19:43:36,909] - [base_gsi:282] INFO - BUILD INDEX on default(employee8998a9981b8442a3b867b941a94d183cjob_title,employee8998a9981b8442a3b867b941a94d183cjoin_yr) USING GSI
[2022-09-10 19:43:37,940] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee8998a9981b8442a3b867b941a94d183cjob_title,employee8998a9981b8442a3b867b941a94d183cjoin_yr) USING GSI
[2022-09-10 19:43:37,967] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee8998a9981b8442a3b867b941a94d183cjob_title%2Cemployee8998a9981b8442a3b867b941a94d183cjoin_yr%29+USING+GSI
[2022-09-10 19:43:38,013] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.123547ms
[2022-09-10 19:43:39,045] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8998a9981b8442a3b867b941a94d183cjob_title'
[2022-09-10 19:43:39,071] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8998a9981b8442a3b867b941a94d183cjob_title%27
[2022-09-10 19:43:39,081] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.929259ms
[2022-09-10 19:43:40,111] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8998a9981b8442a3b867b941a94d183cjob_title'
[2022-09-10 19:43:40,137] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8998a9981b8442a3b867b941a94d183cjob_title%27
[2022-09-10 19:43:40,146] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.243369ms
[2022-09-10 19:43:40,172] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8998a9981b8442a3b867b941a94d183cjoin_yr'
[2022-09-10 19:43:40,198] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8998a9981b8442a3b867b941a94d183cjoin_yr%27
[2022-09-10 19:43:40,201] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.45796ms
[2022-09-10 19:43:41,231] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8998a9981b8442a3b867b941a94d183cjob_title'
[2022-09-10 19:43:41,259] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8998a9981b8442a3b867b941a94d183cjob_title%27
[2022-09-10 19:43:41,268] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.44507ms
[2022-09-10 19:43:41,295] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee8998a9981b8442a3b867b941a94d183cjob_title ON default USING GSI
[2022-09-10 19:43:41,322] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee8998a9981b8442a3b867b941a94d183cjob_title+ON+default+USING+GSI
[2022-09-10 19:43:41,356] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.311991ms
[2022-09-10 19:43:41,399] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8998a9981b8442a3b867b941a94d183cjoin_yr'
[2022-09-10 19:43:41,427] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8998a9981b8442a3b867b941a94d183cjoin_yr%27
[2022-09-10 19:43:41,435] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.608674ms
[2022-09-10 19:43:41,463] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee8998a9981b8442a3b867b941a94d183cjoin_yr ON default USING GSI
[2022-09-10 19:43:41,490] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee8998a9981b8442a3b867b941a94d183cjoin_yr+ON+default+USING+GSI
[2022-09-10 19:43:41,541] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.146766ms
[2022-09-10 19:43:41,575] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8998a9981b8442a3b867b941a94d183cjob_title'
[2022-09-10 19:43:41,602] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8998a9981b8442a3b867b941a94d183cjob_title%27
[2022-09-10 19:43:41,609] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.767443ms
[2022-09-10 19:43:41,609] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd83fe5ea-8132-4d45-b5db-3b34b2afaa41', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.767443ms', 'executionTime': '5.707827ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:43:41,637] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8998a9981b8442a3b867b941a94d183cjoin_yr'
[2022-09-10 19:43:41,662] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8998a9981b8442a3b867b941a94d183cjoin_yr%27
[2022-09-10 19:43:41,665] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.334558ms
[2022-09-10 19:43:41,665] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '376cbe46-9eaa-436e-95df-ad8be812090b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.334558ms', 'executionTime': '1.273187ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:43:41,774] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:43:41,777] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:41,777] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:42,413] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:42,468] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:43:42,469] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:43:42,571] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:43:42,630] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:43:42,633] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:42,634] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:43,294] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:43,354] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:43:43,355] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:43:43,458] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:43:43,515] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:43:43,515] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:43:43,515] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:43:43,540] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:43:43,566] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:43:43,576] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.356957ms
[2022-09-10 19:43:43,604] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:43:43,630] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:43:43,689] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.979454ms
[2022-09-10 19:43:43,755] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:43:43,755] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.92930347352471, 'mem_free': 12893802496, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:43:43,755] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:43:43,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:43,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:44,390] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:44,395] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:44,395] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:45,034] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:45,039] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:45,039] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:46,053] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:46,066] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:46,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:47,151] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:54,071] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-10 19:43:54,216] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:43:54,788] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:43:54,816] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:43:54,816] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:43:54,873] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:43:54,928] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:43:54,983] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:43:54,984] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:43:55,065] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:43:55,067] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:43:55,092] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:43:55,225] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:43:55,225] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:43:55,252] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:43:55,281] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:43:55,281] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:43:55,310] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:43:55,346] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:43:55,346] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:43:55,375] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:43:55,402] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:43:55,402] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:43:55,429] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:43:55,429] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 19:43:55,430] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:43:55,430] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.717s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_8'}
[2022-09-10 19:43:55,512] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:55,512] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:56,162] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:56,195] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:43:56,275] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:43:56,276] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-10 19:43:56,276] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:43:56,539] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:43:56,567] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 13023137792, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:43:56,593] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:43:56,593] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:43:56,594] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:43:56,630] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:43:56,664] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:43:56,665] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:43:56,694] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:43:56,694] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:43:56,695] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:43:56,695] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:43:56,743] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:43:56,746] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:56,747] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:57,382] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:57,383] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:43:57,499] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:43:57,501] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:43:57,529] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:43:57,530] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:43:57,655] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:43:57,682] - [task:165] INFO -  {'uptime': '672', 'memoryTotal': 15466930176, 'memoryFree': 13026459648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:43:57,708] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:43:57,735] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:43:57,735] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:43:57,785] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:43:57,788] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:57,788] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:58,367] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:58,368] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:43:58,481] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:43:58,482] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:43:58,512] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:43:58,513] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:43:58,631] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:43:58,659] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13027094528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:43:58,685] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:43:58,714] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:43:58,714] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:43:58,767] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:43:58,772] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:58,772] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:43:59,419] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:43:59,420] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:43:59,544] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:43:59,545] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:43:59,577] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:43:59,577] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:43:59,702] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:43:59,731] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13027377152, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:43:59,759] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:43:59,790] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:43:59,790] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:43:59,849] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:43:59,852] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:43:59,852] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:44:00,491] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:44:00,492] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:44:00,613] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:44:00,614] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:44:00,645] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:44:00,645] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:44:00,739] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:44:01,391] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:44:06,397] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:44:06,483] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:44:06,487] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:44:06,487] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:44:07,128] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:44:07,130] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:44:07,250] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:44:07,251] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:44:07,252] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:44:07,860] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:44:07,915] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 19:44:07,916] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:44:08,850] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:44:09,182] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:44:09,635] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:44:09,638] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 19:44:09,695] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:44:09,695] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:44:10,277] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:44:10,282] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:44:10,282] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:44:11,085] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:44:11,095] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:44:11,095] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:44:12,199] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:44:12,212] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:44:12,212] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:44:13,319] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:44:20,061] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:44:20,062] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.62526935333998, 'mem_free': 13021478912, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:44:20,062] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:44:20,062] - [basetestcase:467] INFO - Time to execute basesetup : 24.553045988082886
[2022-09-10 19:44:20,113] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:44:20,113] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:44:20,166] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:44:20,166] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:44:20,221] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:44:20,221] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:44:20,274] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:44:20,275] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:44:20,325] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:44:20,391] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:44:20,392] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:44:20,392] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:44:25,404] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:44:25,407] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:44:25,407] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:44:26,038] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:44:27,085] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:44:27,262] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:44:29,457] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:44:29,544] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:44:29,545] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:44:29,545] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:44:59,556] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:44:59,587] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:44:59,613] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:44:59,681] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.914301ms
[2022-09-10 19:44:59,681] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '56ee79c3-c2f1-436a-b2fa-f403c8e80ebf', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.914301ms', 'executionTime': '65.840963ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:44:59,682] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:44:59,707] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:44:59,734] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:45:00,415] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 678.228103ms
[2022-09-10 19:45:00,415] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:45:00,480] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:45:00,515] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:45:00,524] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.758823ms
[2022-09-10 19:45:00,721] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:45:00,756] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:45:00,780] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:45:00,781] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:45:00,792] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:45:00,796] - [base_gsi:326] INFO - []
[2022-09-10 19:45:01,681] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:45:01,709] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:45:01,747] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.615323ms
[2022-09-10 19:45:01,747] - [base_gsi:282] INFO - BUILD INDEX on default(employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr) USING GSI
[2022-09-10 19:45:02,779] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr) USING GSI
[2022-09-10 19:45:02,806] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr%29+USING+GSI
[2022-09-10 19:45:02,832] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.338525ms
[2022-09-10 19:45:03,864] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr'
[2022-09-10 19:45:03,892] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr%27
[2022-09-10 19:45:03,901] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.711854ms
[2022-09-10 19:45:04,931] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr'
[2022-09-10 19:45:04,957] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr%27
[2022-09-10 19:45:04,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.899459ms
[2022-09-10 19:45:05,998] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr'
[2022-09-10 19:45:06,026] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr%27
[2022-09-10 19:45:06,036] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.983618ms
[2022-09-10 19:45:06,064] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr ON default USING GSI
[2022-09-10 19:45:06,090] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr+ON+default+USING+GSI
[2022-09-10 19:45:06,135] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.725546ms
[2022-09-10 19:45:06,170] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr'
[2022-09-10 19:45:06,198] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed17ce65b236a4394b64f10dfd6178052job_title_join_yr%27
[2022-09-10 19:45:06,207] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.794408ms
[2022-09-10 19:45:06,207] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3aaf2686-3d83-44fb-a4e9-c9c6d419e6e9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.794408ms', 'executionTime': '6.670445ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:45:06,319] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:45:06,322] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:06,322] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:07,000] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:07,060] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:45:07,060] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:45:07,177] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:45:07,238] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:45:07,241] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:07,241] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:07,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:07,973] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:45:07,973] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:45:08,083] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:45:08,141] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:45:08,141] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:45:08,142] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:45:08,169] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:45:08,197] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:45:08,207] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.640588ms
[2022-09-10 19:45:08,237] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:45:08,265] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:45:08,309] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.192904ms
[2022-09-10 19:45:08,378] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:45:08,378] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.716592122777371, 'mem_free': 12900188160, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:45:08,378] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:45:08,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:08,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:09,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:09,062] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:09,062] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:10,075] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:10,087] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:10,088] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:11,199] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:11,206] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:11,207] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:12,361] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:18,922] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-10 19:45:19,069] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:45:19,430] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:45:19,458] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:45:19,458] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:45:19,513] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:45:19,564] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:45:19,618] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:45:19,619] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:45:19,698] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:45:19,699] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:45:19,724] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:45:19,854] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:45:19,854] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:45:19,882] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:45:19,909] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:45:19,909] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:45:19,935] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:45:19,962] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:45:19,962] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:45:19,988] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:45:20,013] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:45:20,013] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:45:20,039] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:45:20,040] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 19:45:20,040] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:45:20,040] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.583s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_9'}
[2022-09-10 19:45:20,123] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:20,124] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:20,730] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:20,764] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:45:20,844] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:45:20,845] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-10 19:45:20,845] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:45:21,150] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:45:21,177] - [task:165] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 13006733312, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:45:21,203] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:45:21,203] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:45:21,204] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:45:21,235] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:45:21,271] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:45:21,272] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:45:21,299] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:45:21,300] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:45:21,301] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:45:21,301] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:45:21,353] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:45:21,356] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:21,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:21,973] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:21,974] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:45:22,097] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:45:22,099] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:45:22,129] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:45:22,129] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:45:22,256] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:45:22,284] - [task:165] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 13012107264, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:45:22,310] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:45:22,338] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:45:22,339] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:45:22,390] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:45:22,393] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:22,393] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:23,031] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:23,032] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:45:23,147] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:45:23,149] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:45:23,179] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:45:23,179] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:45:23,295] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:45:23,322] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 13012172800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:45:23,348] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:45:23,374] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:45:23,375] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:45:23,425] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:45:23,431] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:23,431] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:24,043] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:24,044] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:45:24,163] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:45:24,164] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:45:24,195] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:45:24,196] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:45:24,326] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:45:24,353] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 13010825216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:45:24,380] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:45:24,408] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:45:24,408] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:45:24,465] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:45:24,468] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:24,469] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:25,114] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:25,115] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:45:25,236] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:45:25,237] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:45:25,268] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:45:25,269] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:45:25,360] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:45:26,010] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:45:31,016] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:45:31,103] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:45:31,107] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:31,108] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:31,775] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:31,776] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:45:31,893] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:45:31,894] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:45:31,895] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:45:32,490] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:45:32,543] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 19:45:32,544] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:45:33,604] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:45:33,911] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:45:34,175] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:45:34,178] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 19:45:34,232] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:34,232] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:34,854] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:34,860] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:34,860] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:35,726] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:35,735] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:35,735] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:36,875] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:36,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:36,887] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:38,039] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:44,758] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:45:44,759] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.67115439114872, 'mem_free': 13016768512, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:45:44,759] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:45:44,759] - [basetestcase:467] INFO - Time to execute basesetup : 24.63816261291504
[2022-09-10 19:45:44,811] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:45:44,811] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:45:44,863] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:45:44,864] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:45:44,917] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:45:44,918] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:45:44,972] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:45:44,973] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:45:45,024] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:45:45,090] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:45:45,091] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:45:45,091] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:45:50,103] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:45:50,107] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:45:50,108] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:45:50,729] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:45:51,883] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:45:52,062] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:45:54,191] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:45:54,268] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:45:54,269] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:45:54,269] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:46:24,287] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:46:24,317] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:46:24,341] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:46:24,412] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.782986ms
[2022-09-10 19:46:24,412] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '16262cbb-6777-4646-a7fb-72c7086cb8e9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.782986ms', 'executionTime': '68.721409ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:46:24,412] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:46:24,439] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:46:24,465] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:46:25,129] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 661.513269ms
[2022-09-10 19:46:25,129] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:46:25,197] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:46:25,232] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:46:25,240] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.949958ms
[2022-09-10 19:46:25,455] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:46:25,490] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:46:25,510] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:46:25,510] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:46:25,527] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:46:25,609] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:46:26,418] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3131ab55ab6c4d8596f3bbc0967f7000join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:46:26,445] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3131ab55ab6c4d8596f3bbc0967f7000join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:46:26,475] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.323154ms
[2022-09-10 19:46:26,476] - [base_gsi:282] INFO - BUILD INDEX on default(`employee3131ab55ab6c4d8596f3bbc0967f7000join_yr`) USING GSI
[2022-09-10 19:46:27,508] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee3131ab55ab6c4d8596f3bbc0967f7000join_yr`) USING GSI
[2022-09-10 19:46:27,535] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee3131ab55ab6c4d8596f3bbc0967f7000join_yr%60%29+USING+GSI
[2022-09-10 19:46:27,563] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.231762ms
[2022-09-10 19:46:27,616] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3131ab55ab6c4d8596f3bbc0967f7000join_yr'
[2022-09-10 19:46:27,652] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3131ab55ab6c4d8596f3bbc0967f7000join_yr%27
[2022-09-10 19:46:27,662] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.477832ms
[2022-09-10 19:46:28,693] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3131ab55ab6c4d8596f3bbc0967f7000join_yr'
[2022-09-10 19:46:28,719] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3131ab55ab6c4d8596f3bbc0967f7000join_yr%27
[2022-09-10 19:46:28,729] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.975681ms
[2022-09-10 19:46:29,760] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3131ab55ab6c4d8596f3bbc0967f7000join_yr'
[2022-09-10 19:46:29,785] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3131ab55ab6c4d8596f3bbc0967f7000join_yr%27
[2022-09-10 19:46:29,794] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.641784ms
[2022-09-10 19:46:30,826] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:46:30,850] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 19:46:30,855] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.838103ms
[2022-09-10 19:46:30,855] - [base_gsi:504] INFO - {'requestID': '62f161a6-7fb8-42a0-8a73-258efcb011fd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee3131ab55ab6c4d8596f3bbc0967f7000join_yr', 'index_id': '55a371c13ba31827', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.838103ms', 'executionTime': '2.767684ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 19:46:30,856] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:46:30,856] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:46:30,856] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:46:30,857] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:46:30,858] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:46:30,858] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:46:30,858] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:46:30,859] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:46:30,917] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:46:30,958] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:46:30,984] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 19:46:31,181] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 184.325141ms
[2022-09-10 19:46:31,182] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:46:31,182] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:46:33,111] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:46:33,137] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3131ab55ab6c4d8596f3bbc0967f7000join_yr'
[2022-09-10 19:46:33,163] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3131ab55ab6c4d8596f3bbc0967f7000join_yr%27
[2022-09-10 19:46:33,168] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.238834ms
[2022-09-10 19:46:33,168] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9b37d636-46c0-41ea-9ef4-78bbb7402bc0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.238834ms', 'executionTime': '4.156932ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:46:33,270] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:46:33,273] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:33,273] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:33,964] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:34,019] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:46:34,019] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:46:34,122] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:46:34,180] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:46:34,183] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:34,183] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:34,861] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:34,916] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:46:34,916] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:46:35,027] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:46:35,084] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:46:35,085] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:46:35,085] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:46:35,112] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:46:35,139] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:46:35,147] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.679229ms
[2022-09-10 19:46:35,147] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'afb24831-db45-4eb9-80ee-97f2f9317ab2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.679229ms', 'executionTime': '6.616219ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:46:35,202] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:46:35,202] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.33789968538871, 'mem_free': 12858408960, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:46:35,203] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:46:35,206] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:35,206] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:35,850] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:35,855] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:35,855] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:36,476] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:36,480] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:36,481] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:37,382] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:37,390] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:37,390] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:38,534] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:46,075] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-10 19:46:46,208] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:46:46,260] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:46:46,313] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:46:46,365] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:46:46,365] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:46:46,440] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:46:46,442] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:46:46,467] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:46:46,597] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:46:46,597] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:46:46,624] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:46:46,650] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:46:46,650] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:46:46,676] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:46:46,704] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:46:46,704] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:46:46,733] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:46:46,761] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:46:46,761] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:46:46,791] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:46:46,791] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 19:46:46,792] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:46:46,792] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.722s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_10'}
[2022-09-10 19:46:46,873] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:46,873] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:47,534] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:47,568] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:46:47,650] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:46:47,650] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-10 19:46:47,651] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:46:47,900] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:46:47,927] - [task:165] INFO -  {'uptime': '841', 'memoryTotal': 15466930176, 'memoryFree': 13004054528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:46:47,955] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:46:47,955] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:46:47,955] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:46:47,991] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:46:48,027] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:46:48,028] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:46:48,056] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:46:48,058] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:46:48,058] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:46:48,058] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:46:48,112] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:46:48,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:48,116] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:48,752] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:48,753] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:46:48,874] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:46:48,875] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:46:48,907] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:46:48,908] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:46:49,030] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:46:49,057] - [task:165] INFO -  {'uptime': '842', 'memoryTotal': 15466930176, 'memoryFree': 13003636736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:46:49,084] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:46:49,112] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:46:49,113] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:46:49,165] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:46:49,171] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:49,171] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:49,862] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:49,863] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:46:49,994] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:46:49,995] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:46:50,025] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:46:50,026] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:46:50,143] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:46:50,170] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 13003935744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:46:50,196] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:46:50,225] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:46:50,225] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:46:50,277] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:46:50,280] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:50,280] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:50,972] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:50,973] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:46:51,102] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:46:51,104] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:46:51,134] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:46:51,135] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:46:51,256] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:46:51,283] - [task:165] INFO -  {'uptime': '839', 'memoryTotal': 15466930176, 'memoryFree': 13003374592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:46:51,309] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:46:51,337] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:46:51,338] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:46:51,388] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:46:51,391] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:51,392] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:52,033] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:52,034] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:46:52,151] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:46:52,153] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:46:52,182] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:46:52,183] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:46:52,276] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:46:52,798] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:46:57,799] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:46:57,882] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:46:57,895] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:46:57,895] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:46:58,567] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:46:58,568] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:46:58,692] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:46:58,694] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:46:58,694] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:46:59,394] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:46:59,466] - [on_prem_rest_client:3047] INFO - 0.07 seconds to create bucket default
[2022-09-10 19:46:59,467] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:47:00,383] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:47:00,727] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:47:01,051] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:47:01,055] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 19:47:01,128] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:47:01,128] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:47:01,806] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:47:01,810] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:47:01,811] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:47:02,848] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:47:02,856] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:47:02,856] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:47:04,012] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:47:04,024] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:47:04,024] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:47:05,151] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:47:12,090] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:47:12,090] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.74424853437333, 'mem_free': 13000093696, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:47:12,090] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:47:12,090] - [basetestcase:467] INFO - Time to execute basesetup : 25.21987223625183
[2022-09-10 19:47:12,142] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:47:12,143] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:47:12,197] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:47:12,197] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:47:12,250] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:47:12,251] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:47:12,303] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:47:12,303] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:47:12,352] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:47:12,411] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:47:12,413] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:47:12,413] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:47:17,425] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:47:17,430] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:47:17,430] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:47:18,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:47:19,208] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:47:19,382] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:47:21,596] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:47:21,677] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:47:21,677] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:47:21,677] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:47:51,698] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:47:51,728] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:47:51,755] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:47:51,825] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.82969ms
[2022-09-10 19:47:51,826] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b7b8af2e-4dd3-4eca-8c9f-95d229a4a124', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.82969ms', 'executionTime': '68.766137ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:47:51,826] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:47:51,852] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:47:51,877] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:47:52,588] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 708.202794ms
[2022-09-10 19:47:52,588] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:47:52,649] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:47:52,687] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:47:52,695] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.428706ms
[2022-09-10 19:47:52,899] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:47:52,935] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:47:52,950] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:47:52,951] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:47:52,978] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:47:53,058] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:47:53,857] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0a7f51815acc4be7a93ef9a7b6af4471join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:47:53,884] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0a7f51815acc4be7a93ef9a7b6af4471join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:47:53,915] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.120802ms
[2022-09-10 19:47:53,915] - [base_gsi:282] INFO - BUILD INDEX on default(`employee0a7f51815acc4be7a93ef9a7b6af4471join_yr`) USING GSI
[2022-09-10 19:47:54,945] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee0a7f51815acc4be7a93ef9a7b6af4471join_yr`) USING GSI
[2022-09-10 19:47:54,971] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee0a7f51815acc4be7a93ef9a7b6af4471join_yr%60%29+USING+GSI
[2022-09-10 19:47:54,992] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.42102ms
[2022-09-10 19:47:55,039] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0a7f51815acc4be7a93ef9a7b6af4471join_yr'
[2022-09-10 19:47:55,077] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0a7f51815acc4be7a93ef9a7b6af4471join_yr%27
[2022-09-10 19:47:55,086] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.286736ms
[2022-09-10 19:47:56,117] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0a7f51815acc4be7a93ef9a7b6af4471join_yr'
[2022-09-10 19:47:56,144] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0a7f51815acc4be7a93ef9a7b6af4471join_yr%27
[2022-09-10 19:47:56,153] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.583853ms
[2022-09-10 19:47:57,184] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0a7f51815acc4be7a93ef9a7b6af4471join_yr'
[2022-09-10 19:47:57,212] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0a7f51815acc4be7a93ef9a7b6af4471join_yr%27
[2022-09-10 19:47:57,219] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.849101ms
[2022-09-10 19:47:58,250] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:47:58,276] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 19:47:58,280] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.779239ms
[2022-09-10 19:47:58,280] - [base_gsi:504] INFO - {'requestID': '66d3c803-9048-46fb-b4f4-0095697fa64f', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0a7f51815acc4be7a93ef9a7b6af4471join_yr', 'index_id': 'e044d814e2dc5de5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.779239ms', 'executionTime': '2.713104ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 19:47:58,281] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:47:58,281] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:47:58,282] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:47:58,282] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:47:58,282] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:47:58,282] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:47:58,283] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:47:58,283] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:47:58,342] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:47:58,381] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:47:58,407] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 19:47:58,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 165.285204ms
[2022-09-10 19:47:58,586] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:47:58,587] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:47:59,790] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-10 19:47:59,845] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-10 19:47:59,876] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:47:59,907] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 19:47:59,910] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.718682ms
[2022-09-10 19:47:59,910] - [base_gsi:504] INFO - {'requestID': '7ec4a123-ce61-4a10-a251-058d5180a2a0', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0a7f51815acc4be7a93ef9a7b6af4471join_yr', 'index_id': 'e044d814e2dc5de5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.718682ms', 'executionTime': '1.660617ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 19:47:59,911] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:47:59,911] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:47:59,911] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:47:59,912] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:47:59,912] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:47:59,912] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:47:59,913] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:47:59,913] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:47:59,973] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:48:00,013] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:48:00,041] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 19:48:00,143] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 92.206335ms
[2022-09-10 19:48:00,143] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:48:00,144] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:48:01,328] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0a7f51815acc4be7a93ef9a7b6af4471join_yr ON default USING GSI
[2022-09-10 19:48:01,357] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0a7f51815acc4be7a93ef9a7b6af4471join_yr+ON+default+USING+GSI
[2022-09-10 19:48:01,401] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.319941ms
[2022-09-10 19:48:01,434] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0a7f51815acc4be7a93ef9a7b6af4471join_yr'
[2022-09-10 19:48:01,462] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0a7f51815acc4be7a93ef9a7b6af4471join_yr%27
[2022-09-10 19:48:01,470] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.970296ms
[2022-09-10 19:48:01,470] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '71848876-4ec8-4e09-87ab-e235295cd48f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.970296ms', 'executionTime': '5.89908ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:48:01,582] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:48:01,585] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:01,586] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:02,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:02,381] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:48:02,381] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:48:02,495] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:48:02,555] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:48:02,558] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:02,559] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:03,295] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:03,350] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:48:03,351] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:48:03,465] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:48:03,520] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:48:03,520] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:48:03,521] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:48:03,548] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:48:03,574] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:48:03,583] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.412107ms
[2022-09-10 19:48:03,609] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:48:03,636] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:48:03,681] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.735761ms
[2022-09-10 19:48:03,756] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:48:03,756] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.91613599426874, 'mem_free': 12824231936, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:48:03,756] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:48:03,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:03,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:04,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:04,433] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:04,433] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:05,104] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:05,108] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:05,109] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:06,114] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:06,126] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:06,126] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:07,388] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:15,299] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-10 19:48:15,443] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:48:15,930] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:48:15,959] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:48:15,960] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:48:16,014] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:48:16,069] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:48:16,125] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:48:16,126] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:48:16,212] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:48:16,213] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:48:16,240] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:48:16,369] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:48:16,369] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:48:16,394] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:48:16,419] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:48:16,419] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:48:16,445] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:48:16,471] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:48:16,472] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:48:16,498] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:48:16,523] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:48:16,524] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:48:16,551] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:48:16,552] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 19:48:16,552] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:48:16,553] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 89.747s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_11'}
[2022-09-10 19:48:16,648] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:16,648] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:17,381] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:17,414] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:48:17,492] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:48:17,493] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-10 19:48:17,493] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:48:17,674] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:48:17,701] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12988989440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:48:17,726] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:48:17,726] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:48:17,727] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:48:17,758] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:48:17,796] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:48:17,796] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:48:17,825] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:48:17,826] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:48:17,826] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:48:17,826] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:48:17,873] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:48:17,876] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:17,876] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:18,566] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:18,567] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:48:18,698] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:48:18,699] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:48:18,730] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:48:18,731] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:48:18,859] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:48:18,888] - [task:165] INFO -  {'uptime': '933', 'memoryTotal': 15466930176, 'memoryFree': 12988284928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:48:18,914] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:48:18,943] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:48:18,943] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:48:18,994] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:48:18,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:18,998] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:19,645] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:19,646] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:48:19,774] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:48:19,775] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:48:19,804] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:48:19,805] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:48:19,925] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:48:19,951] - [task:165] INFO -  {'uptime': '929', 'memoryTotal': 15466930176, 'memoryFree': 12985335808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:48:19,976] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:48:20,004] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:48:20,004] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:48:20,057] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:48:20,063] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:20,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:20,759] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:20,760] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:48:20,894] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:48:20,895] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:48:20,924] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:48:20,924] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:48:21,058] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:48:21,085] - [task:165] INFO -  {'uptime': '929', 'memoryTotal': 15466930176, 'memoryFree': 12988780544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:48:21,111] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:48:21,140] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:48:21,141] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:48:21,191] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:48:21,194] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:21,194] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:21,901] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:21,903] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:48:22,039] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:48:22,041] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:48:22,072] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:48:22,073] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:48:22,168] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:48:22,725] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:48:27,728] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:48:27,815] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:48:27,820] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:27,820] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:28,511] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:28,512] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:48:28,642] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:48:28,643] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:48:28,644] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:48:29,314] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:48:29,368] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 19:48:29,369] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:48:30,267] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:48:30,586] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:48:30,848] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:48:30,851] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 19:48:30,905] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:30,905] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:31,614] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:31,619] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:31,620] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:32,679] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:32,689] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:32,689] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:33,887] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:33,895] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:33,895] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:35,134] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:42,265] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:48:42,266] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.02003289654586, 'mem_free': 12987748352, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:48:42,266] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:48:42,266] - [basetestcase:467] INFO - Time to execute basesetup : 25.62076759338379
[2022-09-10 19:48:42,323] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:48:42,323] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:48:42,378] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:48:42,378] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:48:42,434] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:48:42,434] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:48:42,488] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:48:42,489] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:48:42,543] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:48:42,603] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:48:42,603] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:48:42,603] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:48:47,610] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:48:47,613] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:48:47,613] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:48:48,303] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:48:49,466] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:48:49,639] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:48:51,834] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:48:51,914] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:48:51,914] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:48:51,915] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:49:21,940] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:49:21,970] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:49:21,996] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:49:22,063] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.237391ms
[2022-09-10 19:49:22,063] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '151c1898-3edf-4d88-99d0-e668ff91281d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.237391ms', 'executionTime': '65.168027ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:49:22,063] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:49:22,089] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:49:22,114] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:49:22,772] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 656.814014ms
[2022-09-10 19:49:22,773] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:49:22,840] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:49:22,875] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:49:22,883] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.607446ms
[2022-09-10 19:49:23,120] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:49:23,164] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:49:23,184] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:49:23,184] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:49:23,207] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:49:23,269] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:49:24,079] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee2f96547009f943d188097862a078fa9ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 19:49:24,106] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee2f96547009f943d188097862a078fa9ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 19:49:24,138] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.29202ms
[2022-09-10 19:49:24,138] - [base_gsi:282] INFO - BUILD INDEX on default(`employee2f96547009f943d188097862a078fa9ajoin_yr`) USING GSI
[2022-09-10 19:49:25,170] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee2f96547009f943d188097862a078fa9ajoin_yr`) USING GSI
[2022-09-10 19:49:25,197] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee2f96547009f943d188097862a078fa9ajoin_yr%60%29+USING+GSI
[2022-09-10 19:49:25,229] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.228864ms
[2022-09-10 19:49:25,272] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2f96547009f943d188097862a078fa9ajoin_yr'
[2022-09-10 19:49:25,304] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2f96547009f943d188097862a078fa9ajoin_yr%27
[2022-09-10 19:49:25,314] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.635973ms
[2022-09-10 19:49:26,345] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2f96547009f943d188097862a078fa9ajoin_yr'
[2022-09-10 19:49:26,371] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2f96547009f943d188097862a078fa9ajoin_yr%27
[2022-09-10 19:49:26,381] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.519157ms
[2022-09-10 19:49:27,416] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2f96547009f943d188097862a078fa9ajoin_yr'
[2022-09-10 19:49:27,443] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2f96547009f943d188097862a078fa9ajoin_yr%27
[2022-09-10 19:49:27,451] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.926708ms
[2022-09-10 19:49:28,482] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:49:28,507] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 19:49:28,511] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.378387ms
[2022-09-10 19:49:28,511] - [base_gsi:504] INFO - {'requestID': '066ea7d5-3460-4b08-aa37-917e1ebe2977', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2f96547009f943d188097862a078fa9ajoin_yr', 'index_id': 'de80612d5c0ab573', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.378387ms', 'executionTime': '2.31176ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 19:49:28,512] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 19:49:28,512] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:49:28,512] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 19:49:28,513] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 19:49:28,513] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:49:28,513] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:49:28,513] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:49:28,514] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 19:49:28,572] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 19:49:28,610] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 19:49:28,635] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 19:49:28,799] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 150.840996ms
[2022-09-10 19:49:28,799] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 19:49:28,800] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 19:49:30,970] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 19:49:33,599] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:49:33,652] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 19:49:33,653] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:49:34,567] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:49:34,905] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:49:35,215] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:49:35,218] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 19:49:37,251] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:49:37,251] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:37,278] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:49:37,307] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:49:37,307] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:37,334] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:49:37,363] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:49:37,364] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:37,392] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:49:37,421] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:49:37,421] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:37,452] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:49:37,511] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:49:37,537] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2f96547009f943d188097862a078fa9ajoin_yr'
[2022-09-10 19:49:37,562] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2f96547009f943d188097862a078fa9ajoin_yr%27
[2022-09-10 19:49:37,605] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.614786ms
[2022-09-10 19:49:37,605] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1be6fd27-00b5-4110-a710-14f9dd0b347d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '40.614786ms', 'executionTime': '40.558196ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:49:37,632] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2f96547009f943d188097862a078fa9ajoin_yr'
[2022-09-10 19:49:37,659] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2f96547009f943d188097862a078fa9ajoin_yr%27
[2022-09-10 19:49:37,661] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.169082ms
[2022-09-10 19:49:37,662] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9a209350-6bdd-446c-9d7f-fcd5b67a2df6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.169082ms', 'executionTime': '1.096175ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:49:37,714] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:49:37,845] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:49:37,847] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:37,847] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:38,572] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:38,632] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:49:38,633] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:49:38,752] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:49:38,810] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:49:38,813] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:38,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:39,557] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:39,616] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:49:39,616] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:49:39,735] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:49:39,795] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:49:39,795] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:49:39,796] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:49:39,821] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:49:39,847] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:49:39,857] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.383235ms
[2022-09-10 19:49:39,857] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f5e30f46-e613-40a7-9826-e81e781f9760', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.383235ms', 'executionTime': '8.315949ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:49:39,857] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:49:39,884] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:49:39,914] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:49:39,917] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.264628ms
[2022-09-10 19:49:39,918] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd0d17b7a-d2f9-42d9-be7e-8d70c2f1c815', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.264628ms', 'executionTime': '1.166052ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:49:39,972] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:49:39,973] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.6173667793855, 'mem_free': 12822118400, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:49:39,973] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:49:39,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:39,976] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:40,710] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:40,715] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:40,715] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:41,485] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:41,492] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:41,492] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:42,764] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:42,772] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:42,772] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:44,025] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:51,599] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 19:49:51,747] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:49:52,063] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:49:52,092] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:49:52,092] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:49:52,149] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:49:52,203] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:49:52,258] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:49:52,258] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:49:52,342] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:49:52,343] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:49:52,370] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:52,507] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:49:52,508] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:52,540] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:49:52,566] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:49:52,567] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:52,596] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:49:52,625] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:49:52,626] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:52,653] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:49:52,678] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:49:52,678] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:52,706] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:49:52,707] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 19:49:52,707] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:49:52,708] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.116s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_19-32-39/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-10 19:49:52,768] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:52,768] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:53,474] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:53,507] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:49:53,588] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 19:49:53,589] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-10 19:49:53,589] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 19:49:53,796] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 19:49:53,824] - [task:165] INFO -  {'uptime': '1030', 'memoryTotal': 15466930176, 'memoryFree': 12979056640, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:49:53,852] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 19:49:53,852] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 19:49:53,852] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 19:49:53,883] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 19:49:53,918] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 19:49:53,919] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 19:49:53,951] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 19:49:53,952] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 19:49:53,952] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 19:49:53,952] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 19:49:54,004] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:49:54,007] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:54,007] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:54,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:54,721] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:49:54,857] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:49:54,859] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:49:54,891] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:49:54,891] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:49:55,017] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 19:49:55,045] - [task:165] INFO -  {'uptime': '1028', 'memoryTotal': 15466930176, 'memoryFree': 12979855360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:49:55,072] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:49:55,100] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 19:49:55,100] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 19:49:55,151] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:49:55,154] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:55,154] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:55,883] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:55,884] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:49:56,019] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:49:56,020] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:49:56,051] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:49:56,051] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:49:56,172] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 19:49:56,200] - [task:165] INFO -  {'uptime': '1029', 'memoryTotal': 15466930176, 'memoryFree': 12970876928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:49:56,226] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:49:56,255] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 19:49:56,256] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 19:49:56,307] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:49:56,310] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:56,310] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:57,068] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:57,069] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:49:57,217] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:49:57,218] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:49:57,251] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:49:57,251] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:49:57,368] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 19:49:57,396] - [task:165] INFO -  {'uptime': '1024', 'memoryTotal': 15466930176, 'memoryFree': 12978159616, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 19:49:57,424] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 19:49:57,454] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 19:49:57,454] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 19:49:57,506] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 19:49:57,509] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:49:57,509] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:49:58,234] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:49:58,235] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:49:58,368] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:49:58,369] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:49:58,398] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 19:49:58,399] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-10 19:49:58,488] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 19:49:59,011] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:50:04,014] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 19:50:04,097] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 19:50:04,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:50:04,102] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:50:04,810] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:50:04,811] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 19:50:04,944] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 19:50:04,945] - [remote_util:5249] INFO - b'ok'
[2022-09-10 19:50:04,946] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 19:50:05,607] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 19:50:05,660] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 19:50:05,661] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 19:50:06,578] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:50:06,886] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:50:07,346] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 19:50:07,348] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-10 19:50:07,403] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:50:07,403] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:50:08,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:50:08,111] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:50:08,111] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:50:09,205] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:50:09,213] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:50:09,213] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:50:10,438] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:50:10,449] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:50:10,449] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:50:11,740] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:50:19,048] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:50:19,049] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.20384775149341, 'mem_free': 12972154880, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:50:19,049] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:50:19,049] - [basetestcase:467] INFO - Time to execute basesetup : 26.28396224975586
[2022-09-10 19:50:19,099] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:50:19,099] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:50:19,152] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:50:19,153] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:50:19,204] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:50:19,204] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:50:19,256] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 19:50:19,256] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 19:50:19,307] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:50:19,369] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-10 19:50:19,369] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 19:50:19,369] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 19:50:24,381] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 19:50:24,385] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:50:24,385] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:50:25,115] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:50:26,190] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 19:50:26,362] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 19:50:29,414] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 19:50:29,501] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:50:29,501] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 19:50:29,501] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 19:50:59,524] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 19:50:59,554] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:50:59,581] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:50:59,650] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.28345ms
[2022-09-10 19:50:59,651] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ab759fd4-9710-41f9-bbf3-d52f10bf4fad', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.28345ms', 'executionTime': '67.218297ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 19:50:59,651] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 19:50:59,679] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 19:50:59,706] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 19:51:00,368] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 660.677762ms
[2022-09-10 19:51:00,369] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 19:51:00,442] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:51:00,481] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:51:00,488] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.820617ms
[2022-09-10 19:51:00,707] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:51:00,756] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 19:51:00,777] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 19:51:00,779] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 19:51:00,797] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 19:51:00,860] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:51:01,009] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:51:01,012] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:51:01,012] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:51:01,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:51:01,896] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:51:01,896] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 19:51:02,028] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:51:02,088] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:51:02,092] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:51:02,092] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:51:02,901] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:51:02,957] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 19:51:02,958] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 19:51:03,089] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 19:51:03,145] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 19:51:03,146] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 19:51:03,146] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:51:03,174] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 19:51:03,200] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 19:51:03,209] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.982257ms
[2022-09-10 19:51:03,235] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 19:51:03,260] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 19:51:03,316] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.030794ms
[2022-09-10 19:51:03,376] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 19:51:03,376] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.694864075831962, 'mem_free': 12802478080, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 19:51:03,377] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 19:51:03,381] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:51:03,381] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:51:04,134] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:51:04,138] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:51:04,139] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:51:05,398] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:51:05,410] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:51:05,410] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:51:06,751] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:51:06,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 19:51:06,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 19:51:08,107] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 19:51:15,944] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-10 19:51:16,092] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 19:51:17,226] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 19:51:17,255] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 19:51:17,255] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 19:51:17,317] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:51:17,371] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:51:17,423] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 19:51:17,424] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 19:51:17,504] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 19:51:17,505] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 19:51:17,530] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:51:17,659] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 19:51:17,659] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:51:17,685] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 19:51:17,711] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 19:51:17,711] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:51:17,738] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 19:51:17,764] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 19:51:17,764] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:51:17,790] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 19:51:17,816] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 19:51:17,816] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 19:51:17,842] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 19:51:17,842] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-10 19:51:17,843] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 19:51:17,843] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.163s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 24505 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-10.09.2022-18.02.cfg
Build Log: make-10.09.2022-18.02.log
Server Log: logs-10.09.2022-18.02.tar.gz

Finished