Building

Started building at 2022/09/11 20:25:30
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 11.09.2022-20.25
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at 0760e59, changes since last good build: 
fatal: Invalid revision range 52396bd6b64d8340b403a6ca18517986458493d9..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable d3674b88] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 49b617e0] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 3e3d5d29] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable c9a4c04f] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable 067e4f3e] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 297b0354] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable d92b91f0] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable 21069dad] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable 61f7f1e9] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 5d28ff3b] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable bb12e4db] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 1d5ceacb] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable e96a08c5] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable bf7bde9] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable d86be15] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 784d076] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 75b5cd8] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 0827eda] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable 33495ee] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable 50ff074] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable fa7256f] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/11 21:23:33
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Sep 11 21:28:58 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 21:32:52 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 21:33:36 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 11 21:35:01 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 11 21:35:53 b" 'stop_on_failure': 'True'}"
Sep 11 21:35:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 21:35:53 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_1'}"
Sep 11 21:35:53 b'-->result: '
Sep 11 21:35:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 11 21:35:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 21:35:53 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_2'}"
Sep 11 21:35:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 21:35:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 11 21:35:53 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_3'}"
Sep 11 21:35:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 21:35:53 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 21:35:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 11 21:35:53 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_4'}"
Sep 11 21:35:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 21:35:53 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 21:35:53 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 21:45:37 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 11 21:46:22 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 11 21:50:06 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 21:54:03 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 21:56:46 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 21:56:46 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_5'}"
Sep 11 21:56:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 21:56:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 11 21:56:46 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 11 21:56:46 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_6'}"
Sep 11 21:56:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 21:56:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 21:56:46 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 11 21:56:46 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_7'}"
Sep 11 21:56:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 21:56:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 21:56:46 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 11 21:56:46 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 11 21:56:46 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-23-54/test_8'}"
Sep 11 21:56:46 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 21:56:46 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 21:56:46 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 21:56:46 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 11 21:56:46 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 11 21:56:47 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 21:56:47 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 21:56:47 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 11 21:56:47 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 11 21:56:47 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 11 21:56:47 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 11 21:56:47 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 11 21:56:47 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 180: go: command not found
/home/buildbot/bin/dotest: line 183: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-11 21:59:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 21:59:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 21:59:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 21:59:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 21:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 21:59:47 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 21:59:59 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 21:59:59 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:00:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-11 22:00:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-11 22:00:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 22:00:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:07 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-11 22:00:08 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 2156179456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:00:08 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:00:09 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1985101824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:00:09 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:00:10 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1833967616, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:00:10 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:00:10 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:00:11 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1759252480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:00:11 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:00:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:00:11 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 22:00:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 22:00:12 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-11 22:00:12 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:00:17 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 22:00:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 22:00:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-11 22:00:18 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 22:00:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 22:00:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 22:00:19 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 22:00:19 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-11 22:00:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 37.64678183569245, 'mem_free': 13423497216, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.637272357940674
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 22:00:23 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 22:00:28 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 22:00:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:00:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:00:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:00:29 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 22:00:29 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 22:00:31 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 22:00:31 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:00:31 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 22:00:31 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 22:01:01 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 22:01:01 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 22:01:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 22:01:01 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 129.112835ms
2022-09-11 22:01:01 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '92935019-9248-419b-8373-008b127f6a02', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '129.112835ms', 'executionTime': '128.979842ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 22:01:01 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 22:01:01 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 22:01:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 720.603873ms
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.149284ms
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 22:01:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.963563ms
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 22:01:03 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 52.367282ms
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.493605236847932, 'mem_free': 13366484992, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 22:01:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 22:01:08 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 22:01:08 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 22:01:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 22:01:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:01:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 22:01:09 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 22:01:09 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 91.095s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.66189646453487, 'mem_free': 13347188736, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:14 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 22:01:14 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:01:14 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:01:14 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:01:14 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:01:14 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 22:01:15 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 22:01:15 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '93', 'memoryTotal': 15466930176, 'memoryFree': 13347188736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 22:01:16 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:01:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13315178496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13347762176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13346197504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 22:01:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-11 22:01:18 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 22:01:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 22:01:24 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 22:01:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 22:01:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-11 22:01:24 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 22:01:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 22:01:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 22:01:26 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 30.7373562207327, 'mem_free': 13331398656, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.19886612892151
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 22:01:30 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:31 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-11 22:01:31 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 22:01:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 22:01:36 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 22:01:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:01:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:01:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:01:37 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 22:01:37 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 22:01:39 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 22:01:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:01:39 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 22:01:39 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 22:02:09 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 22:02:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 22:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 22:02:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 66.667207ms
2022-09-11 22:02:09 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '430c5f85-ef9e-43c8-9975-fe9741f12951', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.667207ms', 'executionTime': '66.516173ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 22:02:09 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 22:02:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 22:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 699.477726ms
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.55129ms
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 22:02:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:02:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-11 22:02:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-11 22:02:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 34.18243ms
2022-09-11 22:02:11 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title) USING GSI
2022-09-11 22:02:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title) USING GSI
2022-09-11 22:02:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title%29+USING+GSI
2022-09-11 22:02:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 18.785075ms
2022-09-11 22:02:13 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title'
2022-09-11 22:02:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title%27
2022-09-11 22:02:13 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.847942ms
2022-09-11 22:02:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title'
2022-09-11 22:02:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title%27
2022-09-11 22:02:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.676763ms
2022-09-11 22:02:15 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 22:02:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 22:02:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-11 22:02:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.230941ms
2022-09-11 22:02:15 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': 'c92362e5-c008-4cef-8b8e-2f0edcf98c84', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title', 'index_id': '608a859c012668a8', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.230941ms', 'executionTime': '2.166429ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-11 22:02:15 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 22:02:15 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 22:02:15 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 22:02:15 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-11 22:02:15 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-11 22:02:15 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-11 22:02:15 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-11 22:02:15 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-11 22:02:15 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-11 22:02:16 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 22:02:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 22:02:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-11 22:02:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 120.699224ms
2022-09-11 22:02:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-11 22:02:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-11 22:02:17 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 22:02:17 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title'
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title%27
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.027683ms
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title ON default USING GSI
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title+ON+default+USING+GSI
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 43.793098ms
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title'
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec07a3b1aef49d3b36ef3a682bc7bc1job_title%27
2022-09-11 22:02:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.125223ms
2022-09-11 22:02:18 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '3663fbcb-40c3-479d-af36-9eebba437448', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.125223ms', 'executionTime': '6.052097ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 22:02:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:02:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:02:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.331374ms
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 22:02:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 43.872426ms
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.420724680359395, 'mem_free': 13263593472, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 22:02:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 22:02:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 22:02:24 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 22:02:25 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:02:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 22:02:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 22:02:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 22:02:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 22:02:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 22:02:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 22:02:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 76.579s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_2'}
[2022-09-11 22:02:26,103] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:26,103] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:26,385] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:26,416] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:02:26,501] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:02:26,502] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-11 22:02:26,502] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:02:27,132] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:02:27,167] - [task:165] INFO -  {'uptime': '164', 'memoryTotal': 15466930176, 'memoryFree': 13227880448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:02:27,193] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:02:27,194] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:02:27,194] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:02:27,231] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:02:27,264] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:02:27,264] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:02:27,294] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:02:27,295] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:02:27,295] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:02:27,296] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:02:27,346] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:02:27,349] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:27,349] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:27,612] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:27,613] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:02:27,680] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:02:27,681] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:02:27,711] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:02:27,712] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:02:27,834] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:02:27,861] - [task:165] INFO -  {'uptime': '160', 'memoryTotal': 15466930176, 'memoryFree': 13260341248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:02:27,886] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:02:27,914] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:02:27,914] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:02:27,964] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:02:27,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:27,968] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:28,222] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:28,223] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:02:28,291] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:02:28,292] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:02:28,323] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:02:28,324] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:02:28,438] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:02:28,465] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13259382784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:02:28,490] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:02:28,517] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:02:28,517] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:02:28,569] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:02:28,573] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:28,573] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:28,834] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:28,835] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:02:28,902] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:02:28,903] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:02:28,933] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:02:28,934] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:02:29,047] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:02:29,073] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13259923456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:02:29,098] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:02:29,124] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:02:29,125] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:02:29,176] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:02:29,179] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:29,179] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:29,440] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:29,441] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:02:29,509] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:02:29,510] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:02:29,541] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:02:29,542] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:02:29,640] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:02:30,158] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:02:35,158] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:02:35,240] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:02:35,245] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:35,245] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:35,521] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:35,522] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:02:35,588] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:02:35,589] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:02:35,589] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:02:36,752] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:02:36,807] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 22:02:36,809] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:02:37,787] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:02:38,091] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:02:38,381] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:02:38,383] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:02:38,438] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:38,439] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:38,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:38,690] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:38,691] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:38,933] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:38,938] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:38,938] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:39,181] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:39,185] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:39,186] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:39,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:43,265] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:02:43,265] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.28928130907615, 'mem_free': 13255356416, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:02:43,265] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:02:43,265] - [basetestcase:467] INFO - Time to execute basesetup : 17.165082693099976
[2022-09-11 22:02:43,319] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:02:43,319] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:02:43,373] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:02:43,373] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:02:43,431] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:02:43,431] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:02:43,483] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:02:43,483] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:02:43,534] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:02:43,592] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:02:43,593] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:02:43,593] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:02:48,602] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:02:48,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:02:48,605] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:02:48,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:02:49,779] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:02:49,947] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:02:51,638] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:02:51,661] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 22:02:52,311] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:02:52,311] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:02:52,312] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:03:22,332] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:03:22,362] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:03:22,390] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:03:22,457] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.084716ms
[2022-09-11 22:03:22,457] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4919f843-58c3-4755-9086-66bda61f8ff2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.084716ms', 'executionTime': '65.026375ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:03:22,457] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:03:22,483] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:03:22,508] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:03:23,147] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 637.067157ms
[2022-09-11 22:03:23,147] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:03:23,200] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:03:23,240] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:03:23,259] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.137334ms
[2022-09-11 22:03:23,447] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:03:23,483] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:03:23,495] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:03:23,495] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:03:23,508] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:03:23,604] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:03:24,410] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee43e078f78bf14c0c859abc0ce672b03ejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:03:24,435] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee43e078f78bf14c0c859abc0ce672b03ejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:03:24,470] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.138754ms
[2022-09-11 22:03:24,471] - [base_gsi:282] INFO - BUILD INDEX on default(employee43e078f78bf14c0c859abc0ce672b03ejob_title) USING GSI
[2022-09-11 22:03:25,500] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee43e078f78bf14c0c859abc0ce672b03ejob_title) USING GSI
[2022-09-11 22:03:25,525] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee43e078f78bf14c0c859abc0ce672b03ejob_title%29+USING+GSI
[2022-09-11 22:03:25,549] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.42134ms
[2022-09-11 22:03:26,579] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee43e078f78bf14c0c859abc0ce672b03ejob_title'
[2022-09-11 22:03:26,605] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee43e078f78bf14c0c859abc0ce672b03ejob_title%27
[2022-09-11 22:03:26,615] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.776669ms
[2022-09-11 22:03:27,646] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee43e078f78bf14c0c859abc0ce672b03ejob_title'
[2022-09-11 22:03:27,673] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee43e078f78bf14c0c859abc0ce672b03ejob_title%27
[2022-09-11 22:03:27,684] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.923761ms
[2022-09-11 22:03:28,125] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 22:03:28,288] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:03:29,313] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:03:29,687] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:03:29,720] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:03:29,745] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 22:03:29,751] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.788649ms
[2022-09-11 22:03:29,751] - [task:3246] INFO - {'requestID': 'e8cd3b53-aa3e-4823-8555-166de78b26c2', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee43e078f78bf14c0c859abc0ce672b03ejob_title', 'index_id': 'e2f2bc0b6a181a94', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.788649ms', 'executionTime': '3.744949ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 22:03:29,751] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:03:29,752] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:03:29,752] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:03:29,752] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:03:29,753] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 22:03:29,753] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:03:29,753] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:03:29,753] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:03:29,754] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:03:30,753] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:03:30,783] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:03:30,809] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 22:03:30,936] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 117.555861ms
[2022-09-11 22:03:30,936] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:03:30,937] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:03:31,717] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:03:31,717] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:03:32,751] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee43e078f78bf14c0c859abc0ce672b03ejob_title'
[2022-09-11 22:03:32,781] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee43e078f78bf14c0c859abc0ce672b03ejob_title%27
[2022-09-11 22:03:32,794] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.756584ms
[2022-09-11 22:03:32,823] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee43e078f78bf14c0c859abc0ce672b03ejob_title ON default USING GSI
[2022-09-11 22:03:32,851] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee43e078f78bf14c0c859abc0ce672b03ejob_title+ON+default+USING+GSI
[2022-09-11 22:03:32,903] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.931724ms
[2022-09-11 22:03:32,938] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee43e078f78bf14c0c859abc0ce672b03ejob_title'
[2022-09-11 22:03:32,966] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee43e078f78bf14c0c859abc0ce672b03ejob_title%27
[2022-09-11 22:03:32,974] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.718969ms
[2022-09-11 22:03:32,974] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '17ab7e13-ea16-4c32-9c43-a637d32a12db', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.718969ms', 'executionTime': '5.655601ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:03:33,089] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:03:33,093] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:33,093] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:33,465] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:33,522] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:03:33,522] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:03:33,584] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:03:33,644] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:03:33,647] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:33,647] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:34,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:34,064] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:03:34,065] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:03:34,126] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:03:34,186] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:03:34,186] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:03:34,186] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:03:34,214] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:03:34,246] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:03:34,253] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.706455ms
[2022-09-11 22:03:34,281] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:03:34,309] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:03:34,350] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.294412ms
[2022-09-11 22:03:34,427] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:03:34,427] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.02658325208352, 'mem_free': 13129601024, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:03:34,427] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:03:34,431] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:34,431] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:34,807] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:34,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:34,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:35,174] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:35,179] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:35,179] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:35,645] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:35,651] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:35,651] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:36,255] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:40,381] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:03:40,526] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:03:41,272] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:03:41,301] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:03:41,302] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:03:41,358] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:03:41,410] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:03:41,466] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:03:41,467] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:03:41,548] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:03:41,549] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:03:41,577] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:03:41,727] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:03:41,727] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:03:41,753] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:03:41,777] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:03:41,778] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:03:41,806] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:03:41,832] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:03:41,833] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:03:41,859] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:03:41,884] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:03:41,885] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:03:41,910] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:03:41,910] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:03:41,911] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:03:41,911] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 75.862s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_3'}
[2022-09-11 22:03:42,009] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:42,009] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:42,303] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:42,334] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:03:42,413] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:03:42,413] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-11 22:03:42,414] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:03:43,039] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:03:43,067] - [task:165] INFO -  {'uptime': '240', 'memoryTotal': 15466930176, 'memoryFree': 13195673600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:03:43,092] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:03:43,093] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:03:43,093] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:03:43,130] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:03:43,167] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:03:43,168] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:03:43,197] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:03:43,198] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:03:43,198] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:03:43,198] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:03:43,250] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:03:43,253] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:43,253] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:43,568] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:43,569] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:03:43,647] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:03:43,648] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:03:43,677] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:03:43,678] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:03:43,806] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:03:43,835] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13202870272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:03:43,861] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:03:43,890] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:03:43,890] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:03:43,944] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:03:43,948] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:43,949] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:44,277] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:44,278] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:03:44,356] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:03:44,358] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:03:44,388] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:03:44,389] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:03:44,513] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:03:44,540] - [task:165] INFO -  {'uptime': '236', 'memoryTotal': 15466930176, 'memoryFree': 13130022912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:03:44,565] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:03:44,593] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:03:44,593] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:03:44,643] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:03:44,648] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:44,648] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:44,950] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:44,951] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:03:45,025] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:03:45,026] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:03:45,055] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:03:45,056] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:03:45,171] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:03:45,198] - [task:165] INFO -  {'uptime': '236', 'memoryTotal': 15466930176, 'memoryFree': 13135888384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:03:45,223] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:03:45,252] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:03:45,252] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:03:45,303] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:03:45,307] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:45,307] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:45,603] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:45,604] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:03:45,684] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:03:45,686] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:03:45,716] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:03:45,717] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:03:45,807] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:03:46,458] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:03:51,459] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:03:51,541] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:03:51,545] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:51,546] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:51,855] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:51,856] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:03:51,930] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:03:51,931] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:03:51,931] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:03:52,961] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:03:53,017] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 22:03:53,018] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:03:53,907] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:03:54,225] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:03:54,497] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:03:54,500] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:03:54,557] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:54,558] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:54,877] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:54,882] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:54,882] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:55,189] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:55,194] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:55,194] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:55,543] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:03:55,551] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:03:55,552] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:03:56,123] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:00,125] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:04:00,126] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.33229434020904, 'mem_free': 13193539584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:04:00,126] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:04:00,126] - [basetestcase:467] INFO - Time to execute basesetup : 18.119444608688354
[2022-09-11 22:04:00,182] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:04:00,182] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:04:00,238] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:04:00,238] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:04:00,297] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:04:00,297] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:04:00,352] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:04:00,352] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:04:00,403] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:04:00,465] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:04:00,465] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:04:00,465] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:04:05,473] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:04:05,477] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:04:05,478] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:04:05,782] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:06,753] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:04:06,916] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:04:09,495] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:04:09,511] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 22:04:10,356] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:04:10,356] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:04:10,356] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:04:40,385] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:04:40,414] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:04:40,439] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:04:40,506] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.67683ms
[2022-09-11 22:04:40,506] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9a7b978b-3622-4e96-9b61-d9408d2a7624', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.67683ms', 'executionTime': '64.61232ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:04:40,506] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:04:40,532] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:04:40,557] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:04:41,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 634.091278ms
[2022-09-11 22:04:41,193] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:04:41,262] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:04:41,303] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:04:41,312] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.105857ms
[2022-09-11 22:04:41,488] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:04:41,521] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:04:41,540] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:04:41,540] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:04:41,556] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:04:41,634] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:04:42,456] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee09bea6ecaab465c9178a58989058e62job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:04:42,481] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee09bea6ecaab465c9178a58989058e62job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:04:42,520] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.911626ms
[2022-09-11 22:04:42,521] - [base_gsi:282] INFO - BUILD INDEX on default(employeee09bea6ecaab465c9178a58989058e62job_title) USING GSI
[2022-09-11 22:04:43,551] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee09bea6ecaab465c9178a58989058e62job_title) USING GSI
[2022-09-11 22:04:43,576] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee09bea6ecaab465c9178a58989058e62job_title%29+USING+GSI
[2022-09-11 22:04:43,603] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.198942ms
[2022-09-11 22:04:44,635] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee09bea6ecaab465c9178a58989058e62job_title'
[2022-09-11 22:04:44,662] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee09bea6ecaab465c9178a58989058e62job_title%27
[2022-09-11 22:04:44,670] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.306738ms
[2022-09-11 22:04:45,700] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee09bea6ecaab465c9178a58989058e62job_title'
[2022-09-11 22:04:45,725] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee09bea6ecaab465c9178a58989058e62job_title%27
[2022-09-11 22:04:45,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.31844ms
[2022-09-11 22:04:46,114] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 22:04:46,369] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:04:47,182] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:04:47,734] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:04:47,767] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:04:47,793] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 22:04:47,797] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.780454ms
[2022-09-11 22:04:47,797] - [task:3246] INFO - {'requestID': '97f8fbe2-5ee4-4135-bc0d-5f210c84e0c7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee09bea6ecaab465c9178a58989058e62job_title', 'index_id': 'efcde70c35fb31c7', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '1.780454ms', 'executionTime': '1.719355ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 22:04:47,797] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:04:47,797] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:04:47,798] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:04:47,798] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:04:47,798] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 22:04:47,799] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:04:47,799] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:04:47,799] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:04:47,799] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:04:48,799] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:04:48,828] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:04:48,854] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 22:04:48,977] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 113.665014ms
[2022-09-11 22:04:48,978] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:04:48,978] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:04:49,770] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:04:49,770] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:04:50,801] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee09bea6ecaab465c9178a58989058e62job_title'
[2022-09-11 22:04:50,827] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee09bea6ecaab465c9178a58989058e62job_title%27
[2022-09-11 22:04:50,838] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.371484ms
[2022-09-11 22:04:50,865] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee09bea6ecaab465c9178a58989058e62job_title ON default USING GSI
[2022-09-11 22:04:50,891] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee09bea6ecaab465c9178a58989058e62job_title+ON+default+USING+GSI
[2022-09-11 22:04:50,930] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.364289ms
[2022-09-11 22:04:50,966] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee09bea6ecaab465c9178a58989058e62job_title'
[2022-09-11 22:04:50,992] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee09bea6ecaab465c9178a58989058e62job_title%27
[2022-09-11 22:04:50,999] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.520457ms
[2022-09-11 22:04:50,999] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '097e7e8e-289f-4659-b7ab-1f7c92b9c2cb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.520457ms', 'executionTime': '5.466195ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:04:51,103] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:04:51,106] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:04:51,107] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:04:51,497] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:51,553] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:04:51,553] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:04:51,620] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:04:51,675] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:04:51,678] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:04:51,678] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:04:52,055] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:52,114] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:04:52,114] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:04:52,182] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:04:52,239] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:04:52,239] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:04:52,239] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:04:52,265] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:04:52,293] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:04:52,303] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.06784ms
[2022-09-11 22:04:52,330] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:04:52,356] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:04:52,399] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.396526ms
[2022-09-11 22:04:52,475] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:04:52,475] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.06286190917664, 'mem_free': 13014077440, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:04:52,476] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:04:52,480] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:04:52,480] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:04:52,899] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:52,904] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:04:52,904] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:04:53,373] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:53,382] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:04:53,382] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:04:54,094] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:54,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:04:54,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:04:54,780] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:04:59,120] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:04:59,266] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:05:00,529] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:05:00,558] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:05:00,558] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:05:00,620] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:05:00,676] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:05:00,729] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:05:00,730] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:05:00,811] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:05:00,812] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:05:00,838] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:05:00,968] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:05:00,969] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:05:00,995] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:05:01,021] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:05:01,021] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:05:01,047] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:05:01,073] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:05:01,073] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:05:01,099] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:05:01,124] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:05:01,124] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:05:01,151] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:05:01,151] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:05:01,152] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:05:01,152] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.199s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_4'}
[2022-09-11 22:05:01,253] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:01,253] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:01,666] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:01,698] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:05:01,785] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:05:01,786] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-11 22:05:01,786] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:05:02,284] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:05:02,314] - [task:165] INFO -  {'uptime': '320', 'memoryTotal': 15466930176, 'memoryFree': 13151469568, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:05:02,342] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:05:02,342] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:05:02,343] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:05:02,373] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:05:02,412] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:05:02,413] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:05:02,443] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:05:02,444] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:05:02,444] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:05:02,445] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:05:02,495] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:05:02,498] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:02,498] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:02,897] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:02,898] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:05:02,984] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:05:02,985] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:05:03,019] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:05:03,020] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:05:03,169] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:05:03,198] - [task:165] INFO -  {'uptime': '316', 'memoryTotal': 15466930176, 'memoryFree': 13081395200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:05:03,224] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:05:03,253] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:05:03,253] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:05:03,305] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:05:03,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:03,308] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:03,680] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:03,681] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:05:03,772] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:05:03,773] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:05:03,805] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:05:03,805] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:05:03,929] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:05:03,957] - [task:165] INFO -  {'uptime': '316', 'memoryTotal': 15466930176, 'memoryFree': 13081645056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:05:03,989] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:05:04,019] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:05:04,019] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:05:04,071] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:05:04,075] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:04,075] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:04,459] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:04,460] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:05:04,545] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:05:04,546] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:05:04,577] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:05:04,578] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:05:04,700] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:05:04,728] - [task:165] INFO -  {'uptime': '316', 'memoryTotal': 15466930176, 'memoryFree': 13152448512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:05:04,757] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:05:04,787] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:05:04,788] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:05:04,841] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:05:04,844] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:04,844] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:05,224] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:05,225] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:05:05,309] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:05:05,311] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:05:05,341] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:05:05,341] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:05:05,429] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:05:05,951] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:05:10,956] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:05:11,049] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:05:11,052] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:11,053] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:11,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:11,457] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:05:11,539] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:05:11,540] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:05:11,540] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:05:12,558] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:05:12,614] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 22:05:12,614] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:05:13,542] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:05:13,844] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:05:14,112] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:05:14,115] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:05:14,168] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:14,168] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:14,544] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:14,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:14,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:14,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:14,961] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:14,961] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:15,553] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:15,563] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:15,564] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:16,249] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:20,682] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:05:20,683] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.22283472173222, 'mem_free': 13150294016, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:05:20,683] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:05:20,683] - [basetestcase:467] INFO - Time to execute basesetup : 19.432694673538208
[2022-09-11 22:05:20,734] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:05:20,734] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:05:20,788] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:05:20,788] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:05:20,839] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:05:20,839] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:05:20,892] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:05:20,892] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:05:20,941] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:05:20,999] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:05:21,000] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:05:21,000] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:05:26,011] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:05:26,015] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:05:26,015] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:05:26,367] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:05:27,343] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:05:27,591] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:05:30,092] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:05:30,108] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 22:05:30,792] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:05:30,793] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:05:30,793] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:06:00,822] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:06:00,851] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:06:00,876] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:06:00,945] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.805127ms
[2022-09-11 22:06:00,945] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '367d0d74-a7c5-4fd8-b3d9-992e9a682279', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.805127ms', 'executionTime': '66.738487ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:06:00,945] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:06:00,970] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:06:00,996] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:06:01,692] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 694.573948ms
[2022-09-11 22:06:01,693] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:06:01,743] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:06:01,794] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:06:01,808] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.888773ms
[2022-09-11 22:06:02,024] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:06:02,064] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:06:02,083] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:06:02,084] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:06:02,093] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:06:02,162] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:06:02,986] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea2873587d29d4acba8cf502100bfef28job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:06:03,017] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeea2873587d29d4acba8cf502100bfef28job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:06:03,051] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.88163ms
[2022-09-11 22:06:03,052] - [base_gsi:282] INFO - BUILD INDEX on default(employeea2873587d29d4acba8cf502100bfef28job_title) USING GSI
[2022-09-11 22:06:04,083] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeea2873587d29d4acba8cf502100bfef28job_title) USING GSI
[2022-09-11 22:06:04,108] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeea2873587d29d4acba8cf502100bfef28job_title%29+USING+GSI
[2022-09-11 22:06:04,130] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.178627ms
[2022-09-11 22:06:05,161] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea2873587d29d4acba8cf502100bfef28job_title'
[2022-09-11 22:06:05,186] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea2873587d29d4acba8cf502100bfef28job_title%27
[2022-09-11 22:06:05,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.130302ms
[2022-09-11 22:06:06,223] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea2873587d29d4acba8cf502100bfef28job_title'
[2022-09-11 22:06:06,248] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea2873587d29d4acba8cf502100bfef28job_title%27
[2022-09-11 22:06:06,255] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.662419ms
[2022-09-11 22:06:06,731] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 22:06:06,895] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:06:07,768] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:06:07,961] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-11 22:06:07,961] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:06:08,152] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-11 22:06:08,152] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-11 22:06:08,153] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (1758664497, 0, b'')
[2022-09-11 22:06:08,258] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:06:08,286] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:06:08,313] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 22:06:08,317] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.24993ms
[2022-09-11 22:06:08,317] - [task:3246] INFO - {'requestID': 'afbb75ef-5d11-4967-8271-96b9f1d4c2a4', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea2873587d29d4acba8cf502100bfef28job_title', 'index_id': 'e4284ab388e95e5a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.24993ms', 'executionTime': '2.168799ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 22:06:08,317] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:06:08,317] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:06:08,318] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:06:08,318] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:06:08,319] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 22:06:08,319] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:06:08,319] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:06:08,319] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:06:08,319] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:06:09,318] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:06:09,353] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:06:09,380] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 22:06:09,524] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 135.594678ms
[2022-09-11 22:06:09,524] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:06:09,525] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:06:10,304] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:06:10,304] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:06:11,335] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea2873587d29d4acba8cf502100bfef28job_title'
[2022-09-11 22:06:11,360] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea2873587d29d4acba8cf502100bfef28job_title%27
[2022-09-11 22:06:11,370] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.38425ms
[2022-09-11 22:06:11,397] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea2873587d29d4acba8cf502100bfef28job_title ON default USING GSI
[2022-09-11 22:06:11,424] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeea2873587d29d4acba8cf502100bfef28job_title+ON+default+USING+GSI
[2022-09-11 22:06:11,461] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.985971ms
[2022-09-11 22:06:11,496] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea2873587d29d4acba8cf502100bfef28job_title'
[2022-09-11 22:06:11,522] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea2873587d29d4acba8cf502100bfef28job_title%27
[2022-09-11 22:06:11,530] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.331689ms
[2022-09-11 22:06:11,531] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '493506c5-8a4d-48cf-8a29-ef845523fe5b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.331689ms', 'executionTime': '6.268697ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:06:11,640] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:06:11,643] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:11,643] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:12,112] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:12,171] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:06:12,171] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:06:12,245] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:06:12,300] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:06:12,303] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:12,303] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:12,733] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:12,793] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:06:12,793] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:06:12,871] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:06:12,925] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:06:12,926] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:06:12,926] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:06:12,951] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:06:12,976] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:06:12,986] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.228717ms
[2022-09-11 22:06:13,013] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:06:13,038] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:06:13,086] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.124623ms
[2022-09-11 22:06:13,152] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:06:13,152] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.07965068602744, 'mem_free': 12991741952, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:06:13,152] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:06:13,157] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:13,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:13,596] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:13,601] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:13,601] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:14,244] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:14,256] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:14,257] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:15,017] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:15,028] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:15,028] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:15,792] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:20,282] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:06:20,425] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:06:21,135] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:06:21,162] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:06:21,163] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:06:21,216] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:06:21,268] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:06:21,321] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:06:21,322] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:06:21,404] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:06:21,405] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:06:21,430] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:06:21,577] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:06:21,577] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:06:21,603] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:06:21,629] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:06:21,629] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:06:21,656] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:06:21,685] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:06:21,685] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:06:21,713] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:06:21,738] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:06:21,738] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:06:21,765] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:06:21,765] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:06:21,765] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:06:21,765] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.565s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_5'}
[2022-09-11 22:06:21,864] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:21,864] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:22,259] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:22,289] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:06:22,368] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:06:22,368] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-11 22:06:22,368] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:06:22,893] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:06:22,920] - [task:165] INFO -  {'uptime': '400', 'memoryTotal': 15466930176, 'memoryFree': 13121789952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:06:22,945] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:06:22,945] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:06:22,946] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:06:22,983] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:06:23,020] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:06:23,020] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:06:23,048] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:06:23,048] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:06:23,048] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:06:23,049] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:06:23,097] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:06:23,100] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:23,100] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:23,514] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:23,515] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:06:23,608] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:06:23,609] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:06:23,641] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:06:23,642] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:06:23,763] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:06:23,790] - [task:165] INFO -  {'uptime': '396', 'memoryTotal': 15466930176, 'memoryFree': 13002526720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:06:23,817] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:06:23,844] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:06:23,845] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:06:23,899] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:06:23,902] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:23,902] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:24,374] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:24,376] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:06:24,472] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:06:24,473] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:06:24,503] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:06:24,504] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:06:24,621] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:06:24,649] - [task:165] INFO -  {'uptime': '397', 'memoryTotal': 15466930176, 'memoryFree': 13007933440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:06:24,676] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:06:24,705] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:06:24,706] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:06:24,758] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:06:24,763] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:24,764] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:25,179] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:25,180] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:06:25,271] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:06:25,272] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:06:25,302] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:06:25,303] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:06:25,424] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:06:25,452] - [task:165] INFO -  {'uptime': '397', 'memoryTotal': 15466930176, 'memoryFree': 13122596864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:06:25,479] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:06:25,507] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:06:25,508] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:06:25,562] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:06:25,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:25,565] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:25,969] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:25,970] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:06:26,059] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:06:26,060] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:06:26,089] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:06:26,090] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:06:26,180] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:06:26,841] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:06:31,846] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:06:31,930] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:06:31,934] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:31,935] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:32,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:32,360] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:06:32,453] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:06:32,454] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:06:32,454] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:06:33,338] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:06:33,391] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 22:06:33,391] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:06:34,272] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:06:34,587] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:06:34,864] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:06:34,867] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:06:34,924] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:34,924] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:35,352] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:35,357] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:35,357] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:35,759] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:35,764] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:35,764] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:36,433] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:36,442] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:36,442] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:37,173] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:42,147] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:06:42,148] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.76715048508837, 'mem_free': 13120217088, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:06:42,148] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:06:42,148] - [basetestcase:467] INFO - Time to execute basesetup : 20.286195278167725
[2022-09-11 22:06:42,204] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:06:42,204] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:06:42,259] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:06:42,260] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:06:42,313] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:06:42,313] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:06:42,367] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:06:42,367] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:06:42,419] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:06:42,476] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:06:42,477] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:06:42,478] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:06:47,490] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:06:47,493] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:06:47,493] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:06:47,888] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:06:48,896] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:06:49,145] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:06:51,571] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:06:51,587] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 22:06:52,643] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:06:52,644] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:06:52,644] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:07:22,664] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:07:22,695] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:07:22,721] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:07:22,790] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.19387ms
[2022-09-11 22:07:22,791] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4f3d5893-6ce8-4916-a448-420a7864ce2a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.19387ms', 'executionTime': '67.132032ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:07:22,791] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:07:22,817] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:07:22,841] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:07:23,519] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 675.958901ms
[2022-09-11 22:07:23,520] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:07:23,571] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:07:23,615] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:07:23,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.220334ms
[2022-09-11 22:07:23,827] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:07:23,860] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:07:23,874] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:07:23,874] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:07:23,896] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:07:23,988] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:07:24,792] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee06a1b900141346b8a98cf3c65eb4bee8job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:07:24,816] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee06a1b900141346b8a98cf3c65eb4bee8job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:07:24,850] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.240535ms
[2022-09-11 22:07:24,851] - [base_gsi:282] INFO - BUILD INDEX on default(employee06a1b900141346b8a98cf3c65eb4bee8job_title) USING GSI
[2022-09-11 22:07:25,881] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee06a1b900141346b8a98cf3c65eb4bee8job_title) USING GSI
[2022-09-11 22:07:25,906] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee06a1b900141346b8a98cf3c65eb4bee8job_title%29+USING+GSI
[2022-09-11 22:07:25,929] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.281485ms
[2022-09-11 22:07:26,963] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06a1b900141346b8a98cf3c65eb4bee8job_title'
[2022-09-11 22:07:26,988] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06a1b900141346b8a98cf3c65eb4bee8job_title%27
[2022-09-11 22:07:26,998] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.52253ms
[2022-09-11 22:07:28,029] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06a1b900141346b8a98cf3c65eb4bee8job_title'
[2022-09-11 22:07:28,055] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06a1b900141346b8a98cf3c65eb4bee8job_title%27
[2022-09-11 22:07:28,064] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.93531ms
[2022-09-11 22:07:28,557] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:07:28,724] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:07:31,441] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:07:32,068] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:07:32,100] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:07:32,127] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 22:07:32,132] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.980835ms
[2022-09-11 22:07:32,132] - [task:3246] INFO - {'requestID': 'd0258a68-d6e8-411c-b8d2-0d9e8201ed20', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee06a1b900141346b8a98cf3c65eb4bee8job_title', 'index_id': '83b941745b9c05d3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.980835ms', 'executionTime': '2.857067ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 22:07:32,133] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:07:32,133] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:07:32,133] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:07:32,134] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:07:32,134] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 22:07:32,134] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:07:32,135] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:07:32,135] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:07:32,135] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:07:33,134] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:07:33,165] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:07:33,195] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 22:07:33,442] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 227.982057ms
[2022-09-11 22:07:33,443] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:07:33,444] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:07:35,039] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:07:35,039] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:07:36,070] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06a1b900141346b8a98cf3c65eb4bee8job_title'
[2022-09-11 22:07:36,097] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06a1b900141346b8a98cf3c65eb4bee8job_title%27
[2022-09-11 22:07:36,107] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.339704ms
[2022-09-11 22:07:36,134] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee06a1b900141346b8a98cf3c65eb4bee8job_title ON default USING GSI
[2022-09-11 22:07:36,162] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee06a1b900141346b8a98cf3c65eb4bee8job_title+ON+default+USING+GSI
[2022-09-11 22:07:36,207] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.307865ms
[2022-09-11 22:07:36,242] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06a1b900141346b8a98cf3c65eb4bee8job_title'
[2022-09-11 22:07:36,268] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06a1b900141346b8a98cf3c65eb4bee8job_title%27
[2022-09-11 22:07:36,276] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.81948ms
[2022-09-11 22:07:36,276] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '35afc485-0651-49a3-b4ae-2e7f48f1e9c6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.81948ms', 'executionTime': '5.760107ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:07:36,380] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:07:36,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:36,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:36,936] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:36,997] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:07:36,997] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:07:37,087] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:07:37,148] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:07:37,152] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:37,152] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:37,716] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:37,776] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:07:37,777] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:07:37,867] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:07:37,925] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:07:37,926] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:07:37,926] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:07:37,953] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:07:37,981] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:07:37,991] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.29828ms
[2022-09-11 22:07:38,019] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:07:38,046] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:07:38,092] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.498163ms
[2022-09-11 22:07:38,156] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:07:38,157] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.30502531301146, 'mem_free': 12966109184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:07:38,157] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:07:38,161] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:38,161] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:38,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:38,725] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:38,725] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:39,263] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:39,267] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:39,268] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:40,182] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:40,190] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:40,190] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:41,089] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:46,905] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:07:47,050] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:07:47,943] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:07:47,973] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:07:47,973] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:07:48,029] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:07:48,082] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:07:48,135] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:07:48,136] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:07:48,229] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:07:48,230] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:07:48,256] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:07:48,384] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:07:48,384] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:07:48,410] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:07:48,436] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:07:48,436] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:07:48,462] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:07:48,488] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:07:48,488] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:07:48,523] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:07:48,549] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:07:48,549] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:07:48,576] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:07:48,577] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:07:48,577] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:07:48,577] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 86.767s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_6'}
[2022-09-11 22:07:48,680] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:48,681] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:49,170] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:49,205] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:07:49,287] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:07:49,287] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-11 22:07:49,288] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:07:49,710] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:07:49,739] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13058822144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:07:49,765] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:07:49,766] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:07:49,766] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:07:49,797] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:07:49,846] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:07:49,846] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:07:49,876] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:07:49,877] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:07:49,877] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:07:49,878] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:07:49,927] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:07:49,930] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:49,930] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:50,459] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:50,460] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:07:50,566] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:07:50,567] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:07:50,599] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:07:50,599] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:07:50,739] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:07:50,768] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13068328960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:07:50,796] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:07:50,825] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:07:50,825] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:07:50,877] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:07:50,881] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:50,881] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:51,391] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:51,392] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:07:51,499] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:07:51,500] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:07:51,533] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:07:51,533] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:07:51,653] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:07:51,682] - [task:165] INFO -  {'uptime': '482', 'memoryTotal': 15466930176, 'memoryFree': 13067767808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:07:51,708] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:07:51,736] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:07:51,736] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:07:51,789] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:07:51,794] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:51,794] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:52,316] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:52,317] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:07:52,421] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:07:52,422] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:07:52,454] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:07:52,455] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:07:52,577] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:07:52,603] - [task:165] INFO -  {'uptime': '482', 'memoryTotal': 15466930176, 'memoryFree': 13067243520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:07:52,629] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:07:52,660] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:07:52,660] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:07:52,711] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:07:52,714] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:52,715] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:53,226] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:53,227] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:07:53,327] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:07:53,328] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:07:53,357] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:07:53,358] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:07:53,448] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:07:53,946] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:07:58,950] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:07:59,035] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:07:59,040] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:07:59,040] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:07:59,584] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:07:59,585] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:07:59,696] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:07:59,698] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:07:59,698] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:08:00,605] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:08:00,662] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 22:08:00,662] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:08:01,567] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:08:01,891] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:08:02,160] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:08:02,163] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:08:02,221] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:08:02,221] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:08:02,721] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:08:02,726] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:08:02,726] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:08:03,344] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:08:03,355] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:08:03,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:08:04,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:08:04,253] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:08:04,254] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:08:05,172] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:08:10,895] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:08:10,895] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.6563939190588, 'mem_free': 13072572416, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:08:10,896] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:08:10,896] - [basetestcase:467] INFO - Time to execute basesetup : 22.218148946762085
[2022-09-11 22:08:10,948] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:08:10,949] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:08:11,005] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:08:11,005] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:08:11,058] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:08:11,058] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:08:11,112] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:08:11,112] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:08:11,164] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:08:11,230] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:08:11,230] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:08:11,230] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:08:16,242] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:08:16,246] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:08:16,247] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:08:16,786] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:08:17,814] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:08:18,081] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:08:20,247] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:08:20,265] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 22:08:21,650] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:08:21,650] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:08:21,650] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:08:51,678] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:08:51,707] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:08:51,734] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:08:51,805] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.175303ms
[2022-09-11 22:08:51,805] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'adba7f49-b02e-4874-bdb2-213aee1e5d32', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.175303ms', 'executionTime': '69.116288ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:08:51,805] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:08:51,832] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:08:51,858] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:08:52,500] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 639.336139ms
[2022-09-11 22:08:52,500] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:08:52,565] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:08:52,608] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:08:52,616] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.621951ms
[2022-09-11 22:08:52,846] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:08:52,882] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:08:52,895] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:08:52,895] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:08:52,912] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:08:52,976] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:08:53,798] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:08:53,824] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:08:53,859] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.942529ms
[2022-09-11 22:08:53,859] - [base_gsi:282] INFO - BUILD INDEX on default(employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title) USING GSI
[2022-09-11 22:08:54,888] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title) USING GSI
[2022-09-11 22:08:54,916] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title%29+USING+GSI
[2022-09-11 22:08:54,937] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.14323ms
[2022-09-11 22:08:55,967] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title'
[2022-09-11 22:08:55,993] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title%27
[2022-09-11 22:08:56,002] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.20331ms
[2022-09-11 22:08:57,033] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title'
[2022-09-11 22:08:57,059] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title%27
[2022-09-11 22:08:57,069] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.051545ms
[2022-09-11 22:08:57,583] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 22:08:57,754] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:08:58,955] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:08:59,593] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 22:08:59,774] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:09:01,004] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:09:01,403] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:09:01,590] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:09:05,231] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:09:06,090] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:09:06,122] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:09:06,148] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 22:09:06,152] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.171988ms
[2022-09-11 22:09:06,153] - [task:3246] INFO - {'requestID': '65bcd03f-a808-482e-8a07-119ded5ba310', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title', 'index_id': 'e48cd1ecec0a1f50', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.171988ms', 'executionTime': '3.047407ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 22:09:06,153] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:09:06,153] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:09:06,154] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:09:06,154] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:09:06,154] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 22:09:06,155] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:09:06,155] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:09:06,155] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:09:06,155] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 22:09:07,154] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:09:07,183] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 22:09:07,209] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 22:09:07,616] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 187.4219ms
[2022-09-11 22:09:07,617] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:09:07,618] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:09:09,180] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:09:09,181] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 22:09:10,212] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title'
[2022-09-11 22:09:10,238] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title%27
[2022-09-11 22:09:10,248] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.821196ms
[2022-09-11 22:09:10,274] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title ON default USING GSI
[2022-09-11 22:09:10,300] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title+ON+default+USING+GSI
[2022-09-11 22:09:10,350] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.990795ms
[2022-09-11 22:09:10,383] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title'
[2022-09-11 22:09:10,408] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeecb7cdf2ab0b4dd8b4ec8b56c696fce8job_title%27
[2022-09-11 22:09:10,415] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.757458ms
[2022-09-11 22:09:10,415] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0b6e7140-de65-42e5-9a5f-b8cc5832496f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.757458ms', 'executionTime': '5.684011ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:09:10,515] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:09:10,518] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:10,518] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:11,155] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:11,212] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:09:11,212] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:09:11,315] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:09:11,370] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:09:11,373] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:11,373] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:11,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:12,042] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:09:12,042] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:09:12,146] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:09:12,201] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:09:12,202] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:09:12,202] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:09:12,227] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:09:12,252] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:09:12,261] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.848436ms
[2022-09-11 22:09:12,287] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:09:12,311] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:09:12,364] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.551555ms
[2022-09-11 22:09:12,443] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:09:12,443] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.17546197060186, 'mem_free': 12878041088, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:09:12,443] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:09:12,447] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:12,447] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:13,067] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:13,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:13,072] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:14,143] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:14,155] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:14,156] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:15,288] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:15,298] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:15,298] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:16,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:22,852] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:09:23,005] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:09:24,109] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:09:24,150] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:09:24,150] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:09:24,219] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:09:24,273] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:09:24,325] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:09:24,326] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:09:24,405] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:09:24,406] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:09:24,432] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:09:24,559] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:09:24,559] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:09:24,586] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:09:24,612] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:09:24,612] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:09:24,638] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:09:24,664] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:09:24,664] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:09:24,690] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:09:24,716] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:09:24,716] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:09:24,742] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:09:24,742] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 22:09:24,743] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:09:24,743] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 96.117s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_7'}
[2022-09-11 22:09:24,875] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:24,876] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:25,442] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:25,474] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:09:25,552] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:09:25,552] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-11 22:09:25,552] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:09:25,902] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:09:25,929] - [task:165] INFO -  {'uptime': '583', 'memoryTotal': 15466930176, 'memoryFree': 13002547200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:09:25,954] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:09:25,954] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:09:25,954] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:09:25,990] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:09:26,023] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:09:26,024] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:09:26,052] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:09:26,052] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:09:26,053] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:09:26,053] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:09:26,100] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:09:26,103] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:26,103] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:26,666] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:26,668] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:09:26,781] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:09:26,782] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:09:26,811] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:09:26,812] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:09:26,931] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:09:26,958] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 13006143488, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:09:26,983] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:09:27,010] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:09:27,010] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:09:27,061] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:09:27,064] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:27,065] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:27,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:27,687] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:09:27,809] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:09:27,811] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:09:27,839] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:09:27,840] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:09:27,956] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:09:27,986] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 13006278656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:09:28,013] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:09:28,044] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:09:28,044] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:09:28,096] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:09:28,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:28,101] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:28,722] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:28,723] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:09:28,840] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:09:28,841] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:09:28,870] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:09:28,870] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:09:28,988] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:09:29,014] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 13005692928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:09:29,039] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:09:29,066] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:09:29,066] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:09:29,118] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:09:29,122] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:29,122] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:29,689] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:29,690] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:09:29,806] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:09:29,808] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:09:29,836] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:09:29,837] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:09:29,925] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:09:30,437] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:09:35,442] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:09:35,527] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:09:35,532] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:35,532] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:36,121] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:36,122] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:09:36,236] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:09:36,237] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:09:36,238] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:09:37,059] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:09:37,117] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 22:09:37,118] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:09:37,999] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:09:38,301] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:09:38,555] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:09:38,558] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 22:09:38,611] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:38,611] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:39,179] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:39,184] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:39,184] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:39,953] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:39,961] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:39,962] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:41,007] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:41,021] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:41,021] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:42,040] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:48,251] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:09:48,252] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.41795796582511, 'mem_free': 13019598848, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:09:48,252] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:09:48,252] - [basetestcase:467] INFO - Time to execute basesetup : 23.379068613052368
[2022-09-11 22:09:48,306] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:09:48,306] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:09:48,361] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:09:48,361] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:09:48,417] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:09:48,418] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:09:48,475] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:09:48,475] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:09:48,527] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:09:48,583] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:09:48,583] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:09:48,584] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:09:53,595] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:09:53,599] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:09:53,599] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:09:54,168] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:09:55,197] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:09:55,377] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:09:57,643] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:09:57,720] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:09:57,721] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:09:57,721] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:10:27,750] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:10:27,780] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:10:27,806] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:10:27,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.27192ms
[2022-09-11 22:10:27,878] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f06fd05c-bdcc-47d0-8160-25d9eb9f21b0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.27192ms', 'executionTime': '69.211769ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:10:27,878] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:10:27,904] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:10:27,931] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:10:28,556] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 622.797685ms
[2022-09-11 22:10:28,557] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:10:28,613] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:10:28,650] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:10:28,660] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.226059ms
[2022-09-11 22:10:28,878] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:10:28,926] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:10:28,944] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:10:28,944] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:10:28,960] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:10:28,962] - [base_gsi:326] INFO - []
[2022-09-11 22:10:29,844] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4984441436f647dcae7084e5252505a7job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:10:29,871] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4984441436f647dcae7084e5252505a7job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:10:29,906] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.028827ms
[2022-09-11 22:10:29,941] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4984441436f647dcae7084e5252505a7join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:10:29,968] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4984441436f647dcae7084e5252505a7join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:10:30,002] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.83272ms
[2022-09-11 22:10:30,002] - [base_gsi:282] INFO - BUILD INDEX on default(employee4984441436f647dcae7084e5252505a7job_title,employee4984441436f647dcae7084e5252505a7join_yr) USING GSI
[2022-09-11 22:10:31,034] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee4984441436f647dcae7084e5252505a7job_title,employee4984441436f647dcae7084e5252505a7join_yr) USING GSI
[2022-09-11 22:10:31,060] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee4984441436f647dcae7084e5252505a7job_title%2Cemployee4984441436f647dcae7084e5252505a7join_yr%29+USING+GSI
[2022-09-11 22:10:31,094] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.399801ms
[2022-09-11 22:10:32,124] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4984441436f647dcae7084e5252505a7job_title'
[2022-09-11 22:10:32,151] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4984441436f647dcae7084e5252505a7job_title%27
[2022-09-11 22:10:32,161] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.225412ms
[2022-09-11 22:10:33,193] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4984441436f647dcae7084e5252505a7job_title'
[2022-09-11 22:10:33,219] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4984441436f647dcae7084e5252505a7job_title%27
[2022-09-11 22:10:33,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.32166ms
[2022-09-11 22:10:33,256] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4984441436f647dcae7084e5252505a7join_yr'
[2022-09-11 22:10:33,284] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4984441436f647dcae7084e5252505a7join_yr%27
[2022-09-11 22:10:33,287] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.797629ms
[2022-09-11 22:10:34,319] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4984441436f647dcae7084e5252505a7job_title'
[2022-09-11 22:10:34,344] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4984441436f647dcae7084e5252505a7job_title%27
[2022-09-11 22:10:34,356] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.675364ms
[2022-09-11 22:10:34,384] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4984441436f647dcae7084e5252505a7job_title ON default USING GSI
[2022-09-11 22:10:34,410] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4984441436f647dcae7084e5252505a7job_title+ON+default+USING+GSI
[2022-09-11 22:10:34,448] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.628619ms
[2022-09-11 22:10:34,492] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4984441436f647dcae7084e5252505a7join_yr'
[2022-09-11 22:10:34,518] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4984441436f647dcae7084e5252505a7join_yr%27
[2022-09-11 22:10:34,526] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.495333ms
[2022-09-11 22:10:34,553] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4984441436f647dcae7084e5252505a7join_yr ON default USING GSI
[2022-09-11 22:10:34,579] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4984441436f647dcae7084e5252505a7join_yr+ON+default+USING+GSI
[2022-09-11 22:10:34,615] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.125887ms
[2022-09-11 22:10:34,652] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4984441436f647dcae7084e5252505a7job_title'
[2022-09-11 22:10:34,680] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4984441436f647dcae7084e5252505a7job_title%27
[2022-09-11 22:10:34,686] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.118309ms
[2022-09-11 22:10:34,687] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1bfd5a50-d448-409d-8359-ee5b79025e13', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.118309ms', 'executionTime': '5.049824ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:10:34,713] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4984441436f647dcae7084e5252505a7join_yr'
[2022-09-11 22:10:34,740] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4984441436f647dcae7084e5252505a7join_yr%27
[2022-09-11 22:10:34,742] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.380351ms
[2022-09-11 22:10:34,743] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '15b4519f-7102-4e90-94d9-f241f154ba7a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.380351ms', 'executionTime': '1.322567ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:10:34,844] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:10:34,847] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:34,847] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:35,435] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:35,493] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:10:35,493] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:10:35,592] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:10:35,649] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:10:35,652] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:35,652] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:36,277] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:36,335] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:10:36,335] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:10:36,433] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:10:36,489] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:10:36,489] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:10:36,489] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:10:36,515] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:10:36,540] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:10:36,551] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.021863ms
[2022-09-11 22:10:36,576] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:10:36,602] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:10:36,659] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.358811ms
[2022-09-11 22:10:36,735] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:10:36,735] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.461698248378587, 'mem_free': 12904583168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:10:36,735] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:10:36,739] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:36,740] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:37,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:37,329] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:37,329] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:37,919] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:37,923] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:37,924] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:38,674] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:38,683] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:38,683] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:39,786] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:46,964] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-11 22:10:47,117] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:10:47,693] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:10:47,722] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:10:47,722] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:10:47,777] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:10:47,830] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:10:47,886] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:10:47,887] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:10:47,965] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:10:47,966] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:10:47,994] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:10:48,123] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:10:48,124] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:10:48,150] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:10:48,175] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:10:48,175] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:10:48,202] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:10:48,234] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:10:48,239] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:10:48,272] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:10:48,298] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:10:48,298] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:10:48,323] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:10:48,324] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 22:10:48,324] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:10:48,324] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 83.502s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_8'}
[2022-09-11 22:10:48,402] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:48,402] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:48,984] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:49,015] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:10:49,094] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:10:49,094] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-11 22:10:49,095] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:10:49,428] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:10:49,456] - [task:165] INFO -  {'uptime': '667', 'memoryTotal': 15466930176, 'memoryFree': 13012660224, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:10:49,482] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:10:49,483] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:10:49,483] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:10:49,520] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:10:49,558] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:10:49,558] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:10:49,589] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:10:49,589] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:10:49,590] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:10:49,590] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:10:49,639] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:10:49,642] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:49,642] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:50,271] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:50,272] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:10:50,389] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:10:50,390] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:10:50,420] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:10:50,420] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:10:50,545] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:10:50,573] - [task:165] INFO -  {'uptime': '667', 'memoryTotal': 15466930176, 'memoryFree': 13021708288, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:10:50,599] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:10:50,627] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:10:50,627] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:10:50,681] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:10:50,684] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:50,684] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:51,317] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:51,318] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:10:51,444] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:10:51,445] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:10:51,478] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:10:51,479] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:10:51,604] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:10:51,632] - [task:165] INFO -  {'uptime': '663', 'memoryTotal': 15466930176, 'memoryFree': 13021937664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:10:51,660] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:10:51,690] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:10:51,690] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:10:51,743] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:10:51,748] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:51,748] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:52,393] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:52,394] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:10:52,519] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:10:52,521] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:10:52,553] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:10:52,554] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:10:52,680] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:10:52,715] - [task:165] INFO -  {'uptime': '663', 'memoryTotal': 15466930176, 'memoryFree': 13020925952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:10:52,743] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:10:52,776] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:10:52,777] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:10:52,831] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:10:52,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:52,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:53,457] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:53,458] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:10:53,571] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:10:53,572] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:10:53,601] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:10:53,602] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:10:53,690] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:10:54,230] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:10:59,235] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:10:59,323] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:10:59,328] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:10:59,329] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:10:59,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:10:59,962] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:11:00,088] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:11:00,089] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:11:00,090] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:11:00,805] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:11:00,858] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 22:11:00,859] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:11:01,761] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:11:02,077] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:11:02,501] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:11:02,503] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 22:11:02,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:11:02,560] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:11:03,203] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:11:03,208] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:11:03,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:11:04,090] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:11:04,099] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:11:04,099] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:11:05,176] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:11:05,183] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:11:05,184] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:11:06,246] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:11:12,374] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:11:12,374] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 28.01673223671613, 'mem_free': 13021913088, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:11:12,374] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:11:12,375] - [basetestcase:467] INFO - Time to execute basesetup : 23.9748318195343
[2022-09-11 22:11:12,427] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:11:12,427] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:11:12,480] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:11:12,480] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:11:12,532] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:11:12,532] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:11:12,584] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:11:12,584] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:11:12,634] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:11:12,694] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:11:12,694] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:11:12,694] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:11:17,706] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:11:17,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:11:17,711] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:11:18,292] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:11:19,316] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:11:19,492] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:11:21,763] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:11:21,846] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:11:21,847] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:11:21,847] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:11:51,859] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:11:51,889] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:11:51,914] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:11:51,985] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.06026ms
[2022-09-11 22:11:51,985] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6416eeaf-ac14-468b-8132-d8fa1b7f41f0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.06026ms', 'executionTime': '67.986039ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:11:51,985] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:11:52,013] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:11:52,039] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:11:52,718] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 677.091129ms
[2022-09-11 22:11:52,719] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:11:52,796] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:11:52,834] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:11:52,844] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.328087ms
[2022-09-11 22:11:53,057] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:11:53,094] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:11:53,124] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:11:53,125] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:11:53,142] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:11:53,145] - [base_gsi:326] INFO - []
[2022-09-11 22:11:54,016] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5c889522403f4f9a99469aaaa291df69job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:11:54,044] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5c889522403f4f9a99469aaaa291df69job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:11:54,087] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.752412ms
[2022-09-11 22:11:54,088] - [base_gsi:282] INFO - BUILD INDEX on default(employee5c889522403f4f9a99469aaaa291df69job_title_join_yr) USING GSI
[2022-09-11 22:11:55,119] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5c889522403f4f9a99469aaaa291df69job_title_join_yr) USING GSI
[2022-09-11 22:11:55,145] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee5c889522403f4f9a99469aaaa291df69job_title_join_yr%29+USING+GSI
[2022-09-11 22:11:55,171] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.549923ms
[2022-09-11 22:11:56,205] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c889522403f4f9a99469aaaa291df69job_title_join_yr'
[2022-09-11 22:11:56,231] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c889522403f4f9a99469aaaa291df69job_title_join_yr%27
[2022-09-11 22:11:56,241] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.750052ms
[2022-09-11 22:11:57,272] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c889522403f4f9a99469aaaa291df69job_title_join_yr'
[2022-09-11 22:11:57,300] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c889522403f4f9a99469aaaa291df69job_title_join_yr%27
[2022-09-11 22:11:57,309] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.021748ms
[2022-09-11 22:11:58,340] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c889522403f4f9a99469aaaa291df69job_title_join_yr'
[2022-09-11 22:11:58,365] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c889522403f4f9a99469aaaa291df69job_title_join_yr%27
[2022-09-11 22:11:58,374] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.266175ms
[2022-09-11 22:11:58,400] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5c889522403f4f9a99469aaaa291df69job_title_join_yr ON default USING GSI
[2022-09-11 22:11:58,424] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5c889522403f4f9a99469aaaa291df69job_title_join_yr+ON+default+USING+GSI
[2022-09-11 22:11:58,467] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.735958ms
[2022-09-11 22:11:58,501] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c889522403f4f9a99469aaaa291df69job_title_join_yr'
[2022-09-11 22:11:58,527] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c889522403f4f9a99469aaaa291df69job_title_join_yr%27
[2022-09-11 22:11:58,533] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.133767ms
[2022-09-11 22:11:58,533] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e12cbf75-5f64-4ecd-a5de-bb63247d0010', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.133767ms', 'executionTime': '5.074031ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:11:58,635] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:11:58,637] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:11:58,638] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:11:59,306] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:11:59,363] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:11:59,364] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:11:59,471] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:11:59,552] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:11:59,555] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:11:59,556] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:00,212] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:00,272] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:12:00,272] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:12:00,381] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:12:00,438] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:12:00,438] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:12:00,439] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:12:00,464] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:12:00,491] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:12:00,502] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.701981ms
[2022-09-11 22:12:00,529] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:12:00,554] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:12:00,599] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.203768ms
[2022-09-11 22:12:00,673] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:12:00,674] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.051474742626286, 'mem_free': 12908417024, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:12:00,674] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:12:00,679] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:00,679] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:01,322] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:01,327] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:01,327] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:02,331] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:02,343] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:02,343] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:03,500] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:03,509] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:03,509] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:04,609] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:11,038] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-11 22:12:11,185] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:12:12,356] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:12:12,384] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:12:12,384] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:12:12,446] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:12:12,499] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:12:12,551] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:12:12,552] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:12:12,630] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:12:12,631] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:12:12,656] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:12:12,790] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:12:12,790] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:12:12,817] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:12:12,842] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:12:12,842] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:12:12,869] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:12:12,893] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:12:12,894] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:12:12,920] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:12:12,945] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:12:12,946] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:12:12,972] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:12:12,972] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 22:12:12,973] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:12:12,973] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.630s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_9'}
[2022-09-11 22:12:13,061] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:13,061] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:13,707] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:13,740] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:12:13,827] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:12:13,827] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-11 22:12:13,828] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:12:14,088] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:12:14,116] - [task:165] INFO -  {'uptime': '751', 'memoryTotal': 15466930176, 'memoryFree': 13029146624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:12:14,142] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:12:14,142] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:12:14,143] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:12:14,176] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:12:14,210] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:12:14,210] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:12:14,240] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:12:14,241] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:12:14,241] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:12:14,241] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:12:14,289] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:12:14,292] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:14,292] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:14,900] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:14,901] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:12:15,019] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:12:15,021] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:12:15,050] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:12:15,051] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:12:15,176] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:12:15,202] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 13028487168, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:12:15,228] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:12:15,256] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:12:15,256] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:12:15,307] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:12:15,311] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:15,311] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:15,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:15,962] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:12:16,090] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:12:16,091] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:12:16,121] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:12:16,121] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:12:16,241] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:12:16,271] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 13021798400, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:12:16,298] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:12:16,329] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:12:16,330] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:12:16,383] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:12:16,388] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:16,388] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:17,005] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:17,006] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:12:17,122] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:12:17,123] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:12:17,157] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:12:17,158] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:12:17,273] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:12:17,301] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 13029851136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:12:17,327] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:12:17,355] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:12:17,356] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:12:17,410] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:12:17,413] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:17,413] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:17,995] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:17,996] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:12:18,111] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:12:18,112] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:12:18,141] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:12:18,142] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:12:18,237] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:12:18,761] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:12:23,767] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:12:23,854] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:12:23,859] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:23,860] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:24,529] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:24,531] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:12:24,663] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:12:24,664] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:12:24,664] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:12:25,357] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:12:25,414] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 22:12:25,414] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:12:26,487] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:12:26,787] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:12:27,051] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:12:27,053] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 22:12:27,110] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:27,110] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:27,753] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:27,758] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:27,758] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:28,679] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:28,687] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:28,687] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:29,806] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:29,814] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:29,814] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:30,925] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:37,586] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:12:37,586] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.58851207994491, 'mem_free': 13024653312, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:12:37,587] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:12:37,587] - [basetestcase:467] INFO - Time to execute basesetup : 24.52817702293396
[2022-09-11 22:12:37,639] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:12:37,639] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:12:37,696] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:12:37,697] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:12:37,751] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:12:37,751] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:12:37,806] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:12:37,806] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:12:37,859] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:12:37,923] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:12:37,923] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:12:37,923] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:12:42,932] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:12:42,935] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:12:42,936] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:12:43,606] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:12:44,772] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:12:44,943] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:12:47,053] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:12:47,135] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:12:47,135] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:12:47,135] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:13:17,164] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:13:17,193] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:13:17,219] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:13:17,286] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.758894ms
[2022-09-11 22:13:17,286] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd95ef333-2a89-4c21-9cea-d9d987669c02', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.758894ms', 'executionTime': '65.693958ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:13:17,287] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:13:17,313] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:13:17,338] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:13:18,052] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 711.976183ms
[2022-09-11 22:13:18,052] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:13:18,133] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:13:18,168] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:13:18,176] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.278051ms
[2022-09-11 22:13:18,369] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:13:18,404] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:13:18,434] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:13:18,434] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:13:18,452] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:13:18,512] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:13:19,327] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee58c6257a0bca49d09ae4ea7275daa449join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:13:19,352] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee58c6257a0bca49d09ae4ea7275daa449join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:13:19,394] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.151007ms
[2022-09-11 22:13:19,394] - [base_gsi:282] INFO - BUILD INDEX on default(`employee58c6257a0bca49d09ae4ea7275daa449join_yr`) USING GSI
[2022-09-11 22:13:20,425] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee58c6257a0bca49d09ae4ea7275daa449join_yr`) USING GSI
[2022-09-11 22:13:20,453] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee58c6257a0bca49d09ae4ea7275daa449join_yr%60%29+USING+GSI
[2022-09-11 22:13:20,475] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.380348ms
[2022-09-11 22:13:20,521] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee58c6257a0bca49d09ae4ea7275daa449join_yr'
[2022-09-11 22:13:20,559] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee58c6257a0bca49d09ae4ea7275daa449join_yr%27
[2022-09-11 22:13:20,568] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.087948ms
[2022-09-11 22:13:21,599] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee58c6257a0bca49d09ae4ea7275daa449join_yr'
[2022-09-11 22:13:21,624] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee58c6257a0bca49d09ae4ea7275daa449join_yr%27
[2022-09-11 22:13:21,632] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.206372ms
[2022-09-11 22:13:22,662] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee58c6257a0bca49d09ae4ea7275daa449join_yr'
[2022-09-11 22:13:22,688] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee58c6257a0bca49d09ae4ea7275daa449join_yr%27
[2022-09-11 22:13:22,698] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.344764ms
[2022-09-11 22:13:23,729] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:13:23,754] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 22:13:23,758] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.722969ms
[2022-09-11 22:13:23,759] - [base_gsi:504] INFO - {'requestID': '0121b831-6315-475f-bd44-18147fd2558b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee58c6257a0bca49d09ae4ea7275daa449join_yr', 'index_id': '61f39e87901674c9', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.722969ms', 'executionTime': '2.655484ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 22:13:23,759] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:13:23,760] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:13:23,760] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:13:23,760] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:13:23,761] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:13:23,762] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:13:23,762] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:13:23,762] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:13:23,820] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:13:23,861] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:13:23,887] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 22:13:24,056] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 157.819591ms
[2022-09-11 22:13:24,057] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:13:24,057] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:13:25,938] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:13:25,964] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee58c6257a0bca49d09ae4ea7275daa449join_yr'
[2022-09-11 22:13:25,988] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee58c6257a0bca49d09ae4ea7275daa449join_yr%27
[2022-09-11 22:13:25,994] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.302068ms
[2022-09-11 22:13:25,995] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5d163a97-d408-441d-9b09-03495b6bc6c1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.302068ms', 'executionTime': '4.22368ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:13:26,095] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:13:26,098] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:26,098] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:26,810] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:26,864] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:13:26,864] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:13:26,968] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:13:27,025] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:13:27,028] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:27,028] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:27,650] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:27,705] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:13:27,706] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:13:27,811] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:13:27,865] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:13:27,866] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:13:27,866] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:13:27,891] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:13:27,916] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:13:27,923] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.586795ms
[2022-09-11 22:13:27,923] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'af090e4e-e897-408c-b34d-77ab1955021c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.586795ms', 'executionTime': '5.538287ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:13:27,974] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:13:27,974] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.57015424684972, 'mem_free': 12882644992, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:13:27,975] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:13:27,978] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:27,978] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:28,642] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:28,647] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:28,647] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:29,794] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:29,805] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:29,806] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:31,003] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:31,012] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:31,012] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:32,180] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:38,545] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-11 22:13:38,676] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:13:38,729] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:13:38,781] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:13:38,833] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:13:38,833] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:13:38,908] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:13:38,909] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:13:38,933] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:13:39,062] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:13:39,062] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:13:39,088] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:13:39,114] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:13:39,114] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:13:39,140] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:13:39,166] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:13:39,166] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:13:39,192] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:13:39,217] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:13:39,217] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:13:39,243] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:13:39,243] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 22:13:39,243] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:13:39,244] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.236s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_10'}
[2022-09-11 22:13:39,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:39,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:39,972] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:40,005] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:13:40,083] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:13:40,083] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-11 22:13:40,084] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:13:40,350] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:13:40,378] - [task:165] INFO -  {'uptime': '835', 'memoryTotal': 15466930176, 'memoryFree': 12869976064, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:13:40,404] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:13:40,404] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:13:40,405] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:13:40,440] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:13:40,471] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:13:40,471] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:13:40,500] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:13:40,500] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:13:40,500] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:13:40,501] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:13:40,548] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:13:40,551] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:40,551] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:41,188] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:41,189] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:13:41,317] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:13:41,318] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:13:41,348] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:13:41,348] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:13:41,473] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:13:41,503] - [task:165] INFO -  {'uptime': '838', 'memoryTotal': 15466930176, 'memoryFree': 13009571840, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:13:41,530] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:13:41,560] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:13:41,560] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:13:41,613] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:13:41,619] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:41,619] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:42,309] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:42,311] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:13:42,439] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:13:42,440] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:13:42,469] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:13:42,470] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:13:42,586] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:13:42,612] - [task:165] INFO -  {'uptime': '834', 'memoryTotal': 15466930176, 'memoryFree': 13009289216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:13:42,637] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:13:42,664] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:13:42,665] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:13:42,717] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:13:42,720] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:42,720] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:43,361] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:43,362] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:13:43,484] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:13:43,485] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:13:43,514] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:13:43,515] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:13:43,638] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:13:43,665] - [task:165] INFO -  {'uptime': '833', 'memoryTotal': 15466930176, 'memoryFree': 13009809408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:13:43,691] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:13:43,719] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:13:43,720] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:13:43,770] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:13:43,774] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:43,774] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:44,423] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:44,424] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:13:44,556] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:13:44,557] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:13:44,590] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:13:44,591] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:13:44,685] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:13:45,210] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:13:50,215] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:13:50,298] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:13:50,302] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:50,302] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:50,948] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:50,949] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:13:51,080] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:13:51,081] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:13:51,081] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:13:51,864] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:13:51,922] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 22:13:51,923] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:13:52,835] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:13:53,157] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:13:53,442] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:13:53,444] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 22:13:53,504] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:53,504] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:54,206] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:54,211] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:54,211] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:55,258] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:55,269] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:55,270] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:56,474] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:13:56,483] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:13:56,484] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:13:57,687] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:14:04,559] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:14:04,559] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.84951544558447, 'mem_free': 13006114816, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:14:04,560] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:14:04,560] - [basetestcase:467] INFO - Time to execute basesetup : 25.23858141899109
[2022-09-11 22:14:04,613] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:14:04,613] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:14:04,669] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:14:04,669] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:14:04,724] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:14:04,724] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:14:04,778] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:14:04,778] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:14:04,830] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:14:04,895] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:14:04,896] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:14:04,900] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:14:09,908] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:14:09,911] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:14:09,912] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:14:10,549] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:14:11,713] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:14:11,879] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:14:13,993] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:14:14,067] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:14:14,068] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:14:14,068] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:14:44,085] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:14:44,114] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:14:44,140] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:14:44,210] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.509572ms
[2022-09-11 22:14:44,210] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4c1d4cf0-7ac7-411f-883c-9efc5a6ebcf5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.509572ms', 'executionTime': '68.442549ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:14:44,211] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:14:44,237] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:14:44,262] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:14:44,933] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 668.681774ms
[2022-09-11 22:14:44,933] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:14:44,994] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:14:45,026] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:14:45,035] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.998944ms
[2022-09-11 22:14:45,242] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:14:45,274] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:14:45,294] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:14:45,296] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:14:45,307] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:14:45,392] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:14:46,203] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:14:46,229] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:14:46,267] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.873073ms
[2022-09-11 22:14:46,268] - [base_gsi:282] INFO - BUILD INDEX on default(`employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr`) USING GSI
[2022-09-11 22:14:47,300] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr`) USING GSI
[2022-09-11 22:14:47,326] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr%60%29+USING+GSI
[2022-09-11 22:14:47,357] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.433667ms
[2022-09-11 22:14:47,395] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr'
[2022-09-11 22:14:47,425] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr%27
[2022-09-11 22:14:47,434] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.941875ms
[2022-09-11 22:14:48,464] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr'
[2022-09-11 22:14:48,489] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr%27
[2022-09-11 22:14:48,499] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.923784ms
[2022-09-11 22:14:49,529] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr'
[2022-09-11 22:14:49,555] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr%27
[2022-09-11 22:14:49,564] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.430275ms
[2022-09-11 22:14:50,595] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:14:50,620] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 22:14:50,625] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.61215ms
[2022-09-11 22:14:50,625] - [base_gsi:504] INFO - {'requestID': 'ec45184d-3c8e-4178-9fcd-0b69058f7c5b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr', 'index_id': '20338288ad753b71', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.61215ms', 'executionTime': '3.53668ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 22:14:50,626] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:14:50,626] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:14:50,627] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:14:50,627] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:14:50,627] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:14:50,627] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:14:50,628] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:14:50,628] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:14:50,687] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:14:50,726] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:14:50,752] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 22:14:50,923] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 159.315122ms
[2022-09-11 22:14:50,924] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:14:50,924] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:14:52,144] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-11 22:14:52,196] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-11 22:14:52,228] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:14:52,254] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 22:14:52,258] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.892575ms
[2022-09-11 22:14:52,258] - [base_gsi:504] INFO - {'requestID': 'dc07124d-95c8-4963-b2b1-636653ca813b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr', 'index_id': '20338288ad753b71', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.892575ms', 'executionTime': '1.822029ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 22:14:52,258] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:14:52,259] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:14:52,259] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:14:52,259] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:14:52,259] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:14:52,259] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:14:52,260] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:14:52,260] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:14:52,314] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:14:52,353] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:14:52,379] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 22:14:52,485] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.768185ms
[2022-09-11 22:14:52,486] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:14:52,486] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:14:53,695] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr ON default USING GSI
[2022-09-11 22:14:53,722] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr+ON+default+USING+GSI
[2022-09-11 22:14:53,770] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.256874ms
[2022-09-11 22:14:53,805] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr'
[2022-09-11 22:14:53,832] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdc65afdf1a84893b0eaaec6e85c3c47join_yr%27
[2022-09-11 22:14:53,842] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.2913ms
[2022-09-11 22:14:53,842] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '08a01928-16c3-42dd-b3b9-c1e012134600', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.2913ms', 'executionTime': '8.216058ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:14:53,951] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:14:53,956] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:14:53,956] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:14:54,647] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:14:54,704] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:14:54,705] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:14:54,813] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:14:54,869] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:14:54,872] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:14:54,872] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:14:55,538] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:14:55,592] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:14:55,592] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:14:55,710] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:14:55,765] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:14:55,765] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:14:55,765] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:14:55,791] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:14:55,816] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:14:55,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.639657ms
[2022-09-11 22:14:55,855] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:14:55,880] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:14:55,951] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.713159ms
[2022-09-11 22:14:56,013] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:14:56,013] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.67718960007767, 'mem_free': 12830416896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:14:56,013] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:14:56,017] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:14:56,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:14:56,681] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:14:56,685] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:14:56,685] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:14:57,357] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:14:57,361] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:14:57,362] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:14:58,275] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:14:58,283] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:14:58,284] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:14:59,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:07,527] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-11 22:15:07,670] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:15:08,523] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:15:08,551] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:15:08,551] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:15:08,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:15:08,662] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:15:08,717] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:15:08,718] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:15:08,814] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:15:08,815] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:15:08,843] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:15:08,985] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:15:08,985] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:15:09,014] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:15:09,040] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:15:09,041] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:15:09,067] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:15:09,095] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:15:09,095] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:15:09,123] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:15:09,150] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:15:09,151] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:15:09,179] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:15:09,179] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 22:15:09,179] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:15:09,180] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 89.912s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_11'}
[2022-09-11 22:15:09,264] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:09,264] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:10,032] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:10,065] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:15:10,146] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:15:10,146] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-11 22:15:10,147] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:15:10,295] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:15:10,325] - [task:165] INFO -  {'uptime': '927', 'memoryTotal': 15466930176, 'memoryFree': 12977586176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:15:10,352] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:15:10,352] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:15:10,353] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:15:10,390] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:15:10,429] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:15:10,430] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:15:10,460] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:15:10,461] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:15:10,461] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:15:10,461] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:15:10,513] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:15:10,516] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:10,516] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:11,236] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:11,237] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:15:11,370] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:15:11,372] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:15:11,403] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:15:11,404] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:15:11,548] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:15:11,580] - [task:165] INFO -  {'uptime': '924', 'memoryTotal': 15466930176, 'memoryFree': 12978728960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:15:11,607] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:15:11,638] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:15:11,638] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:15:11,690] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:15:11,693] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:11,693] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:12,379] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:12,380] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:15:12,505] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:15:12,506] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:15:12,536] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:15:12,537] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:15:12,656] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:15:12,683] - [task:165] INFO -  {'uptime': '924', 'memoryTotal': 15466930176, 'memoryFree': 12978110464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:15:12,709] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:15:12,740] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:15:12,741] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:15:12,797] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:15:12,802] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:12,802] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:13,477] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:13,478] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:15:13,611] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:15:13,612] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:15:13,641] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:15:13,642] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:15:13,766] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:15:13,794] - [task:165] INFO -  {'uptime': '924', 'memoryTotal': 15466930176, 'memoryFree': 12978839552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:15:13,820] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:15:13,849] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:15:13,849] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:15:13,901] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:15:13,904] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:13,904] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:14,604] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:14,605] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:15:14,734] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:15:14,735] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:15:14,763] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:15:14,764] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:15:14,852] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:15:15,526] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:15:20,531] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:15:20,616] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:15:20,621] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:20,621] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:21,302] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:21,303] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:15:21,428] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:15:21,429] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:15:21,429] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:15:22,008] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:15:22,075] - [on_prem_rest_client:3047] INFO - 0.07 seconds to create bucket default
[2022-09-11 22:15:22,075] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:15:22,973] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:15:23,287] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:15:23,546] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:15:23,548] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 22:15:23,604] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:23,604] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:24,278] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:24,283] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:24,283] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:25,299] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:25,307] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:25,307] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:26,496] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:26,508] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:26,508] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:27,692] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:34,484] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:15:34,485] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.05296871187485, 'mem_free': 12974161920, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:15:34,485] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:15:34,485] - [basetestcase:467] INFO - Time to execute basesetup : 25.224070072174072
[2022-09-11 22:15:34,537] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:15:34,538] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:15:34,594] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:15:34,594] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:15:34,648] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:15:34,649] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:15:34,706] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:15:34,706] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:15:34,762] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:15:34,822] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:15:34,822] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:15:34,822] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:15:39,834] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:15:39,838] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:15:39,838] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:15:40,566] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:15:41,749] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:15:41,917] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:15:44,900] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:15:44,974] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:15:44,975] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:15:44,975] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:16:14,988] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:16:15,017] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:16:15,043] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:16:15,112] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.273371ms
[2022-09-11 22:16:15,112] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7461a362-7287-4ce9-aaa1-17b45f53d738', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.273371ms', 'executionTime': '67.195647ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:16:15,112] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:16:15,139] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:16:15,163] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:16:15,863] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 697.15304ms
[2022-09-11 22:16:15,863] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:16:15,937] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:16:15,982] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:16:15,991] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.583134ms
[2022-09-11 22:16:16,180] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:16:16,218] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:16:16,236] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:16:16,237] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:16:16,256] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:16:16,330] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:16:17,146] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 22:16:17,176] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 22:16:17,216] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.131259ms
[2022-09-11 22:16:17,216] - [base_gsi:282] INFO - BUILD INDEX on default(`employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr`) USING GSI
[2022-09-11 22:16:18,246] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr`) USING GSI
[2022-09-11 22:16:18,272] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr%60%29+USING+GSI
[2022-09-11 22:16:18,293] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.504697ms
[2022-09-11 22:16:18,337] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr'
[2022-09-11 22:16:18,370] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr%27
[2022-09-11 22:16:18,379] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.471612ms
[2022-09-11 22:16:19,410] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr'
[2022-09-11 22:16:19,435] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr%27
[2022-09-11 22:16:19,444] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.820716ms
[2022-09-11 22:16:20,476] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr'
[2022-09-11 22:16:20,503] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr%27
[2022-09-11 22:16:20,510] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.7026ms
[2022-09-11 22:16:21,540] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:16:21,565] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 22:16:21,569] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.633187ms
[2022-09-11 22:16:21,570] - [base_gsi:504] INFO - {'requestID': '8d40826e-cac2-433b-8d36-9d9609230034', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr', 'index_id': 'd7dcc9962ef3596c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.633187ms', 'executionTime': '2.567677ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 22:16:21,570] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 22:16:21,570] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:16:21,571] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 22:16:21,571] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 22:16:21,571] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:16:21,571] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:16:21,572] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:16:21,572] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 22:16:21,630] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 22:16:21,669] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 22:16:21,694] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 22:16:21,881] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 172.908369ms
[2022-09-11 22:16:21,881] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 22:16:21,882] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 22:16:23,568] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 22:16:26,646] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:16:26,697] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 22:16:26,698] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:16:27,603] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:16:27,923] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:16:28,200] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:16:28,202] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 22:16:30,235] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:16:30,235] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:30,264] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:16:30,293] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:16:30,294] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:30,323] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:16:30,352] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:16:30,353] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:30,385] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:16:30,415] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:16:30,415] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:30,443] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:16:30,496] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:16:30,521] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr'
[2022-09-11 22:16:30,547] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr%27
[2022-09-11 22:16:30,594] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.475149ms
[2022-09-11 22:16:30,595] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bd5ac9e7-a713-4a8a-ae77-b48dffe012b3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '45.475149ms', 'executionTime': '45.405266ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:16:30,626] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr'
[2022-09-11 22:16:30,652] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5b16f8b59b954b2e80f9f6b29dfeac55join_yr%27
[2022-09-11 22:16:30,655] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.127326ms
[2022-09-11 22:16:30,655] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1cb89549-f6bd-4ddd-984a-4975084255d5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.127326ms', 'executionTime': '1.060243ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:16:30,708] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:16:30,838] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:16:30,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:30,841] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:31,513] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:31,569] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:16:31,570] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:16:31,682] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:16:31,739] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:16:31,742] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:31,742] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:32,484] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:32,539] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:16:32,539] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:16:32,656] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:16:32,715] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:16:32,715] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:16:32,715] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:16:32,742] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:16:32,770] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:16:32,779] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.235837ms
[2022-09-11 22:16:32,779] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5d063e2d-44eb-4428-a267-ccd0396991ae', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.235837ms', 'executionTime': '7.175531ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:16:32,779] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:16:32,806] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:16:32,832] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:16:32,834] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.171414ms
[2022-09-11 22:16:32,834] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fa50f124-d03e-4950-9f37-acf7ab6ac7d6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.171414ms', 'executionTime': '1.104292ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:16:32,886] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:16:32,886] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.30763031102238, 'mem_free': 12964491264, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:16:32,887] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:16:32,890] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:32,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:33,577] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:33,581] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:33,582] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:34,365] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:34,379] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:34,379] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:35,600] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:35,608] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:35,608] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:36,870] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:44,546] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 22:16:44,701] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:16:45,025] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:16:45,055] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:16:45,055] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:16:45,112] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:16:45,166] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:16:45,220] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:16:45,221] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:16:45,301] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:16:45,302] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:16:45,327] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:45,458] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:16:45,458] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:45,485] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:16:45,512] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:16:45,512] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:45,539] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:16:45,565] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:16:45,565] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:45,592] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:16:45,617] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:16:45,617] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:45,643] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:16:45,643] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 22:16:45,644] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:16:45,644] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.440s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_21-59-37/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-11 22:16:45,707] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:45,708] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:46,399] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:46,432] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:16:46,514] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 22:16:46,515] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-11 22:16:46,515] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 22:16:46,733] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 22:16:46,762] - [task:165] INFO -  {'uptime': '1024', 'memoryTotal': 15466930176, 'memoryFree': 12963684352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:16:46,787] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 22:16:46,787] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 22:16:46,788] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 22:16:46,824] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 22:16:46,859] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 22:16:46,860] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 22:16:46,887] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 22:16:46,889] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 22:16:46,889] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 22:16:46,889] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 22:16:46,939] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:16:46,941] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:46,942] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:47,643] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:47,644] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:16:47,777] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:16:47,778] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:16:47,809] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:16:47,810] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:16:47,937] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 22:16:47,964] - [task:165] INFO -  {'uptime': '1024', 'memoryTotal': 15466930176, 'memoryFree': 12963131392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:16:47,992] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:16:48,021] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 22:16:48,021] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 22:16:48,078] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:16:48,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:48,081] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:48,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:48,827] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:16:48,969] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:16:48,970] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:16:49,000] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:16:49,001] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:16:49,124] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 22:16:49,154] - [task:165] INFO -  {'uptime': '1019', 'memoryTotal': 15466930176, 'memoryFree': 12962754560, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:16:49,182] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:16:49,214] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 22:16:49,215] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 22:16:49,270] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:16:49,272] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:49,273] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:50,016] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:50,018] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:16:50,152] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:16:50,153] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:16:50,183] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:16:50,183] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:16:50,298] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 22:16:50,325] - [task:165] INFO -  {'uptime': '1019', 'memoryTotal': 15466930176, 'memoryFree': 12963311616, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 22:16:50,350] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 22:16:50,379] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 22:16:50,379] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 22:16:50,429] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 22:16:50,432] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:50,432] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:51,126] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:51,127] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:16:51,258] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:16:51,259] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:16:51,288] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 22:16:51,289] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-11 22:16:51,378] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 22:16:52,010] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:16:57,015] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 22:16:57,103] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 22:16:57,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:16:57,115] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:16:57,802] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:16:57,803] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 22:16:57,929] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 22:16:57,930] - [remote_util:5249] INFO - b'ok'
[2022-09-11 22:16:57,930] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 22:16:58,515] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 22:16:58,571] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 22:16:58,572] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 22:16:59,486] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:16:59,826] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:17:00,335] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 22:17:00,338] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-11 22:17:00,398] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:00,398] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:01,128] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:01,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:01,132] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:02,347] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:02,359] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:02,360] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:03,612] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:03,619] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:03,619] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:04,877] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:11,916] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:17:11,917] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.74363083360042, 'mem_free': 12961734656, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:17:11,917] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:17:11,917] - [basetestcase:467] INFO - Time to execute basesetup : 26.212554216384888
[2022-09-11 22:17:11,969] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:17:11,969] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:17:12,022] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:17:12,022] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:17:12,075] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:17:12,076] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:17:12,132] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 22:17:12,132] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 22:17:12,184] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:17:12,241] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-11 22:17:12,241] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 22:17:12,241] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 22:17:17,253] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 22:17:17,257] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:17,258] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:17,976] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:19,047] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 22:17:19,214] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 22:17:22,284] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 22:17:22,366] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:17:22,366] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 22:17:22,367] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 22:17:52,394] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 22:17:52,424] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:17:52,451] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:17:52,522] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.542784ms
[2022-09-11 22:17:52,522] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5fead098-5ac9-4530-af22-4a32c5ba1312', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.542784ms', 'executionTime': '68.484718ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 22:17:52,522] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 22:17:52,550] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 22:17:52,574] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 22:17:53,221] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 645.173262ms
[2022-09-11 22:17:53,222] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 22:17:53,275] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:17:53,314] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:17:53,322] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.995632ms
[2022-09-11 22:17:53,512] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:17:53,546] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 22:17:53,568] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 22:17:53,569] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 22:17:53,586] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 22:17:53,664] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:17:53,803] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:17:53,806] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:53,806] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:54,569] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:54,627] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:17:54,627] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 22:17:54,759] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:17:54,816] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:17:54,819] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:54,819] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:55,581] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:55,635] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 22:17:55,635] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 22:17:55,760] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 22:17:55,817] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 22:17:55,817] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 22:17:55,818] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:17:55,843] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 22:17:55,868] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 22:17:55,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.559826ms
[2022-09-11 22:17:55,903] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 22:17:55,928] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 22:17:55,995] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.767925ms
[2022-09-11 22:17:56,057] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 22:17:56,057] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.020674505982688, 'mem_free': 12803055616, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 22:17:56,057] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 22:17:56,062] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:56,062] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:56,798] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:56,803] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:56,803] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:57,983] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:57,992] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:57,992] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:17:59,365] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:17:59,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 22:17:59,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 22:18:00,748] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 22:18:08,636] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-11 22:18:08,778] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 22:18:09,134] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 22:18:09,162] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 22:18:09,162] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 22:18:09,218] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:18:09,270] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:18:09,322] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 22:18:09,323] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 22:18:09,401] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 22:18:09,402] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 22:18:09,427] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:18:09,558] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 22:18:09,558] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:18:09,586] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 22:18:09,611] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 22:18:09,611] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:18:09,638] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 22:18:09,664] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 22:18:09,664] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:18:09,696] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 22:18:09,725] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 22:18:09,725] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 22:18:09,752] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 22:18:09,753] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-11 22:18:09,753] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 22:18:09,753] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 84.128s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 10122 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-11.09.2022-20.25.cfg
Build Log: make-11.09.2022-20.25.log
Server Log: logs-11.09.2022-20.25.tar.gz

Finished