Building

Started building at 2022/09/10 20:03:54
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 10.09.2022-20.03
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at de1130b, changes since last good build: none
Switching plasma to unstable
plasma is at a338d0e, changes since last good build: 
fatal: Invalid revision range 70ba222be48a88c87394d7fa22445b17df4a5ec1..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable 45ec966b] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 8f69f896] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 3d8ef46f] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable 433a8a00] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable a716c699] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable f8fdc1c9] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable 2105e8d8] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable 4e72e8db] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable c63646f1] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 0effc888] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable 5099aebd] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 91b224d3] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179227 at refs/changes/27/179227/10
Total 8 (delta 3), reused 4 (delta 3)
[unstable bc52c5b5] MB-51825: Remove vbuckets arg from indexer command line and its usage
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Aug 25 18:17:32 2022 +0530
 2 files changed, 9 deletions(-)
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable a3fae864] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 17795d3] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable 86875c0] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 108a88a] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable b152ee1] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable daaa633] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable a7a3aca] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable 2549e0a] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 315c2d1] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Pulling in uncommitted change 179769 at refs/changes/69/179769/2
Total 5 (delta 4), reused 5 (delta 4)
[unstable c608a1a] MB-52096: Use memory stats to estimate mutation quota if page stats not available
 Author: jliang00 
 Date: Thu Sep 8 16:56:16 2022 -0700
 3 files changed, 86 insertions(+), 27 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/10 20:57:23
Testing mode: sanity,unit,functional,integration
Using storage type: fdb
Setting ulimit to 200000

Simple Test

Sep 10 21:02:48 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 21:06:43 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 10 21:07:27 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 10 21:08:30 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 10 21:09:17 b" 'stop_on_failure': 'True'}"
Sep 10 21:09:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 21:09:17 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_1'}"
Sep 10 21:09:17 b'-->result: '
Sep 10 21:09:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 10 21:09:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 10 21:09:17 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_2'}"
Sep 10 21:09:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 21:09:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 10 21:09:17 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_3'}"
Sep 10 21:09:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 21:09:17 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 21:09:17 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 10 21:09:17 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_4'}"
Sep 10 21:09:17 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 21:09:17 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 21:09:17 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 21:19:13 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 10 21:19:59 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 10 21:23:43 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 21:27:40 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 21:30:24 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 10 21:30:24 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_5'}"
Sep 10 21:30:24 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 21:30:24 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 10 21:30:24 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 10 21:30:24 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_6'}"
Sep 10 21:30:24 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 21:30:24 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 21:30:24 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 10 21:30:24 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_7'}"
Sep 10 21:30:24 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 21:30:24 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 21:30:24 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 10 21:30:24 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 10 21:30:24 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_20-57-45/test_8'}"
Sep 10 21:30:24 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 10 21:30:24 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 10 21:30:24 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 10 21:30:24 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 10 21:30:24 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 10 21:30:25 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 21:30:25 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 10 21:30:25 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 10 21:30:25 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 10 21:30:25 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 10 21:30:25 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 10 21:30:25 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 10 21:30:25 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=forestdb
Printing gsi_type=forestdb
gsi_type=forestdb
In here
-p makefile=True,gsi_type=forestdb
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'forestdb',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-10 21:33:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:33:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:33:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:33:13 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 21:33:16 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 21:33:22 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:33:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:33:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:33:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:33:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:33:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:33:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 21:34:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:34:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-10 21:34:02 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '45', 'memoryTotal': 15466930176, 'memoryFree': 13593255936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:34:02 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 21:34:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:34:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:34:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '43', 'memoryTotal': 15466930176, 'memoryFree': 13573705728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:34:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 21:34:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:34:04 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:34:04 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '44', 'memoryTotal': 15466930176, 'memoryFree': 13603844096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:34:04 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:34:05 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:34:05 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '39', 'memoryTotal': 15466930176, 'memoryFree': 13578530816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:34:05 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:34:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:34:06 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:34:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:34:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:34:06 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:34:06 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 21:34:06 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 21:34:06 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-10 21:34:06 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 21:34:11 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 21:34:11 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-10 21:34:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:34:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:34:12 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:34:12 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 21:34:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 21:34:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-10 21:34:12 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 21:34:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 21:34:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 21:34:13 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 21:34:13 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 40.61590081244326, 'mem_free': 13435940864, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 64.15928888320923
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 21:34:17 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 21:34:22 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 21:34:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:23 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 21:34:24 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 21:34:25 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 21:34:25 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:34:25 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 21:34:25 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 135.851897ms
2022-09-10 21:34:56 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '4cd17683-c4ba-4aed-b231-d015046e507f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '135.851897ms', 'executionTime': '135.729573ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 715.068817ms
2022-09-10 21:34:56 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.836905ms
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.969013ms
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 47.631857ms
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 10.37783304544111, 'mem_free': 13299552256, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:34:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:34:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:34:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-10 21:35:03 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 21:35:04 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 21:35:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 111.417s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-10 21:35:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 21:35:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 10.37783304544111, 'mem_free': 13299552256, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 21:35:10 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-10 21:35:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-10 21:35:11 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-10 21:35:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '113', 'memoryTotal': 15466930176, 'memoryFree': 13326880768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:35:11 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-10 21:35:11 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-10 21:35:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-10 21:35:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-10 21:35:12 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:35:12 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '114', 'memoryTotal': 15466930176, 'memoryFree': 13331668992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:35:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:35:12 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '110', 'memoryTotal': 15466930176, 'memoryFree': 13331279872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:35:13 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '109', 'memoryTotal': 15466930176, 'memoryFree': 13330640896, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:35:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:35:14 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:35:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-10 21:35:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-10 21:35:14 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-10 21:35:14 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-10 21:35:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 21:35:19 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-10 21:35:19 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-10 21:35:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-10 21:35:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-10 21:35:20 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-10 21:35:20 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-10 21:35:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-10 21:35:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-10 21:35:20 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-10 21:35:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 21:35:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 21:35:21 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-10 21:35:21 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 21:35:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.10534102180791, 'mem_free': 13323177984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.2914400100708
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-10 21:35:26 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-10 21:35:31 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-10 21:35:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:35:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:35:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:35:32 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-10 21:35:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-10 21:35:34 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-10 21:35:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:35:34 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-10 21:35:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-10 21:36:04 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-10 21:36:04 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 21:36:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 21:36:04 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 70.928305ms
2022-09-10 21:36:04 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '02389395-d015-47ca-807e-6e70741d0356', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.928305ms', 'executionTime': '70.798902ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 21:36:04 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-10 21:36:04 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-10 21:36:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 710.006245ms
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.216938ms
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-10 21:36:05 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:36:06 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-10 21:36:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-10 21:36:06 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.376578ms
2022-09-10 21:36:06 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title) USING GSI
2022-09-10 21:36:07 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title) USING GSI
2022-09-10 21:36:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title%29+USING+GSI
2022-09-10 21:36:07 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.173658ms
2022-09-10 21:36:08 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title'
2022-09-10 21:36:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title%27
2022-09-10 21:36:08 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.077852ms
2022-09-10 21:36:09 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title'
2022-09-10 21:36:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title%27
2022-09-10 21:36:10 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.805813ms
2022-09-10 21:36:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 21:36:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 21:36:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-10 21:36:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.306253ms
2022-09-10 21:36:11 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '129e4707-37d5-4d21-9350-771c0d71e9fb', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title', 'index_id': 'f17aa1b3c9fd1b11', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '4.306253ms', 'executionTime': '4.247193ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-10 21:36:11 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 21:36:11 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 21:36:11 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 21:36:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-10 21:36:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-10 21:36:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-10 21:36:11 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-10 21:36:11 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-10 21:36:11 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-10 21:36:12 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 21:36:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-10 21:36:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-10 21:36:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 123.967712ms
2022-09-10 21:36:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-10 21:36:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-10 21:36:13 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 21:36:13 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title'
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title%27
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.789046ms
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title ON default USING GSI
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title+ON+default+USING+GSI
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 39.128684ms
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title'
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee124ab2dfbff24e059cf5aa7f8f4d95aejob_title%27
2022-09-10 21:36:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.8618ms
2022-09-10 21:36:14 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '5a4a1649-e82a-491f-a7fb-c4bb70d8fa74', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.8618ms', 'executionTime': '5.741344ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:36:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.682123ms
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 54.048871ms
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.88085532393624, 'mem_free': 13223755776, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:36:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:36:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:36:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:36:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:36:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:36:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-10 21:36:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-10 21:36:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-10 21:36:20 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 21:36:20 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-10 21:36:20 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-10 21:36:20 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-10 21:36:20 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-10 21:36:20 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:36:20 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-10 21:36:21 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-10 21:36:21 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 76.728s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_2'}
[2022-09-10 21:36:21,540] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:21,540] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:21,802] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:21,833] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:36:21,912] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:36:21,912] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-10 21:36:21,913] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:36:22,569] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:36:22,595] - [task:165] INFO -  {'uptime': '185', 'memoryTotal': 15466930176, 'memoryFree': 13238448128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:36:22,620] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:36:22,621] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:36:22,621] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:36:22,653] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:36:22,690] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:36:22,690] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:36:22,718] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:36:22,718] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:36:22,719] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:36:22,719] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:36:22,768] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:36:22,771] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:22,771] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:23,032] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:23,033] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:36:23,099] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:36:23,100] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:36:23,131] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:36:23,132] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:36:23,158] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:36:23,243] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:36:23,269] - [task:165] INFO -  {'uptime': '184', 'memoryTotal': 15466930176, 'memoryFree': 13210443776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:36:23,295] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:36:23,324] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:36:23,324] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:36:23,376] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:36:23,379] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:23,379] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:23,657] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:23,658] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:36:23,729] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:36:23,730] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:36:23,760] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:36:23,761] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:36:23,786] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:36:23,872] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:36:23,901] - [task:165] INFO -  {'uptime': '180', 'memoryTotal': 15466930176, 'memoryFree': 13210951680, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:36:23,929] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:36:23,960] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:36:23,960] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:36:24,010] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:36:24,014] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:24,014] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:24,303] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:24,304] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:36:24,375] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:36:24,377] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:36:24,407] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:36:24,407] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:36:24,433] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:36:24,520] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:36:24,547] - [task:165] INFO -  {'uptime': '180', 'memoryTotal': 15466930176, 'memoryFree': 13209681920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:36:24,574] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:36:24,605] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:36:24,605] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:36:24,659] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:36:24,663] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:24,663] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:24,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:24,957] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:36:25,031] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:36:25,032] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:36:25,064] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:36:25,065] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:36:25,093] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:36:25,154] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:36:25,754] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:36:30,760] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:36:30,846] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:36:30,851] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:30,852] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:31,140] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:31,141] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:36:31,211] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:36:31,212] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:36:31,212] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:36:32,283] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:36:32,339] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:36:32,339] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:36:33,339] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:36:33,651] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:36:33,990] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:36:33,992] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:36:34,055] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:34,055] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:34,304] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:34,309] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:34,309] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:34,550] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:34,555] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:34,556] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:34,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:34,814] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:34,814] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:35,174] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:38,893] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:36:38,893] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 28.68525277786001, 'mem_free': 13232799744, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:36:38,893] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:36:38,894] - [basetestcase:467] INFO - Time to execute basesetup : 17.356059074401855
[2022-09-10 21:36:38,946] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:36:38,947] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:36:39,008] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:36:39,008] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:36:39,062] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:36:39,062] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:36:39,116] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:36:39,116] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:36:39,168] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:36:39,224] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:36:39,225] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:36:39,225] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:36:44,238] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:36:44,241] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:36:44,241] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:36:44,491] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:36:45,424] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:36:45,596] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:36:48,252] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:36:48,274] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 21:36:48,910] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:36:48,910] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:36:48,910] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:37:18,925] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:37:18,955] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:37:18,979] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:37:19,050] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.131219ms
[2022-09-10 21:37:19,050] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '14ebef12-9d9a-4e45-b875-9cc1fb83956f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.131219ms', 'executionTime': '68.96697ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:37:19,051] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:37:19,076] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:37:19,100] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:37:19,784] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 682.120573ms
[2022-09-10 21:37:19,785] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:37:19,855] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:37:19,893] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:37:19,902] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.747259ms
[2022-09-10 21:37:20,103] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:37:20,143] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:37:20,159] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:37:20,159] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:37:20,184] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:37:20,255] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:37:21,062] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed87429da816747e289b1b156c7bb4e13job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:37:21,089] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed87429da816747e289b1b156c7bb4e13job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:37:21,140] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.101442ms
[2022-09-10 21:37:21,140] - [base_gsi:282] INFO - BUILD INDEX on default(employeed87429da816747e289b1b156c7bb4e13job_title) USING GSI
[2022-09-10 21:37:22,171] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed87429da816747e289b1b156c7bb4e13job_title) USING GSI
[2022-09-10 21:37:22,197] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed87429da816747e289b1b156c7bb4e13job_title%29+USING+GSI
[2022-09-10 21:37:22,224] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.076268ms
[2022-09-10 21:37:23,255] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed87429da816747e289b1b156c7bb4e13job_title'
[2022-09-10 21:37:23,280] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed87429da816747e289b1b156c7bb4e13job_title%27
[2022-09-10 21:37:23,291] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.732545ms
[2022-09-10 21:37:24,322] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed87429da816747e289b1b156c7bb4e13job_title'
[2022-09-10 21:37:24,346] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed87429da816747e289b1b156c7bb4e13job_title%27
[2022-09-10 21:37:24,355] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.638144ms
[2022-09-10 21:37:24,792] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 21:37:24,961] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:37:25,942] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:37:26,358] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:37:26,391] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:37:26,419] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 21:37:26,425] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.629034ms
[2022-09-10 21:37:26,425] - [task:3246] INFO - {'requestID': 'bb0e27f7-a150-4fd0-ad3e-d29b8878c830', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeed87429da816747e289b1b156c7bb4e13job_title', 'index_id': '59cff7e1673a5e83', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.629034ms', 'executionTime': '3.504227ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 21:37:26,425] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:37:26,426] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:37:26,426] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:37:26,426] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:37:26,427] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 21:37:26,427] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:37:26,427] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:37:26,427] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:37:26,428] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:37:27,427] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:37:27,457] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:37:27,484] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 21:37:27,615] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 121.346259ms
[2022-09-10 21:37:27,615] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:37:27,616] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:37:28,368] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:37:28,368] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:37:29,398] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed87429da816747e289b1b156c7bb4e13job_title'
[2022-09-10 21:37:29,424] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed87429da816747e289b1b156c7bb4e13job_title%27
[2022-09-10 21:37:29,433] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.199004ms
[2022-09-10 21:37:29,458] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed87429da816747e289b1b156c7bb4e13job_title ON default USING GSI
[2022-09-10 21:37:29,482] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed87429da816747e289b1b156c7bb4e13job_title+ON+default+USING+GSI
[2022-09-10 21:37:29,531] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.375153ms
[2022-09-10 21:37:29,565] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed87429da816747e289b1b156c7bb4e13job_title'
[2022-09-10 21:37:29,595] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed87429da816747e289b1b156c7bb4e13job_title%27
[2022-09-10 21:37:29,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.056229ms
[2022-09-10 21:37:29,604] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '50aef4e2-cf1f-4ead-805d-414877a0c56d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.056229ms', 'executionTime': '6.997573ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:37:29,714] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:37:29,717] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:29,717] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:30,091] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:30,147] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:37:30,147] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:37:30,214] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:37:30,270] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:37:30,274] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:30,274] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:30,663] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:30,721] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:37:30,722] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:37:30,785] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:37:30,841] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:37:30,842] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:37:30,842] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:37:30,868] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:37:30,894] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:37:30,902] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.054868ms
[2022-09-10 21:37:30,929] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:37:30,954] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:37:31,014] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.593482ms
[2022-09-10 21:37:31,077] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:37:31,077] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.47702441574566, 'mem_free': 13120077824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:37:31,077] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:37:31,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:31,082] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:31,418] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:31,423] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:31,423] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:31,755] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:31,761] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:31,761] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:32,155] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:32,161] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:32,161] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:32,753] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:36,957] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:37:37,109] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:37:37,816] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:37:37,844] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:37:37,845] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:37:37,904] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:37:37,957] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:37:38,008] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:37:38,009] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:37:38,090] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:37:38,091] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:37:38,117] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:37:38,263] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:37:38,264] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:37:38,291] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:37:38,318] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:37:38,318] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:37:38,345] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:37:38,371] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:37:38,371] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:37:38,398] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:37:38,423] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:37:38,424] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:37:38,450] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:37:38,450] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:37:38,451] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:37:38,451] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 76.965s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_3'}
[2022-09-10 21:37:38,550] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:38,550] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:38,852] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:38,882] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:37:38,964] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:37:38,964] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-10 21:37:38,965] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:37:39,579] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:37:39,607] - [task:165] INFO -  {'uptime': '262', 'memoryTotal': 15466930176, 'memoryFree': 13164908544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:37:39,633] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:37:39,633] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:37:39,633] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:37:39,663] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:37:39,697] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:37:39,698] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:37:39,726] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:37:39,727] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:37:39,727] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:37:39,727] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:37:39,775] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:37:39,778] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:39,779] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:40,116] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:40,117] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:37:40,195] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:37:40,196] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:37:40,232] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:37:40,233] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:37:40,259] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:37:40,345] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:37:40,373] - [task:165] INFO -  {'uptime': '259', 'memoryTotal': 15466930176, 'memoryFree': 13174358016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:37:40,399] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:37:40,428] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:37:40,428] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:37:40,480] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:37:40,483] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:40,483] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:40,817] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:40,818] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:37:40,893] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:37:40,894] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:37:40,924] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:37:40,925] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:37:40,951] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:37:41,036] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:37:41,063] - [task:165] INFO -  {'uptime': '260', 'memoryTotal': 15466930176, 'memoryFree': 13175984128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:37:41,089] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:37:41,118] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:37:41,118] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:37:41,168] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:37:41,173] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:41,173] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:41,485] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:41,486] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:37:41,561] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:37:41,562] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:37:41,591] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:37:41,592] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:37:41,617] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:37:41,700] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:37:41,726] - [task:165] INFO -  {'uptime': '255', 'memoryTotal': 15466930176, 'memoryFree': 13174870016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:37:41,751] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:37:41,783] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:37:41,784] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:37:41,835] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:37:41,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:41,839] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:42,143] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:42,144] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:37:42,224] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:37:42,225] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:37:42,257] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:37:42,257] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:37:42,282] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:37:42,338] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:37:42,874] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:37:47,875] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:37:47,962] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:37:47,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:47,967] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:48,286] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:48,287] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:37:48,362] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:37:48,363] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:37:48,363] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:37:49,451] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:37:49,506] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:37:49,507] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:37:50,416] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:37:50,715] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:37:50,973] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:37:50,976] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:37:51,029] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:51,029] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:51,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:51,341] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:51,341] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:51,633] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:51,638] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:51,638] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:51,945] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:51,951] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:37:51,951] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:37:52,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:37:56,505] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:37:56,506] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.48599829536461, 'mem_free': 13177356288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:37:56,506] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:37:56,506] - [basetestcase:467] INFO - Time to execute basesetup : 17.958919763565063
[2022-09-10 21:37:56,558] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:37:56,558] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:37:56,612] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:37:56,612] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:37:56,665] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:37:56,666] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:37:56,719] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:37:56,719] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:37:56,772] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:37:56,837] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:37:56,838] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:37:56,838] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:38:01,850] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:38:01,854] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:01,854] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:02,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:03,147] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:38:03,316] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:38:05,911] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:38:05,929] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 21:38:06,754] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:38:06,754] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:38:06,755] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:38:36,785] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:38:36,815] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:38:36,841] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:38:36,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.869219ms
[2022-09-10 21:38:36,912] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '768d67d2-5cc8-4545-9cc7-6cb59e65ddec', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.869219ms', 'executionTime': '68.731627ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:38:36,912] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:38:36,938] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:38:36,964] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:38:37,662] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 696.115288ms
[2022-09-10 21:38:37,663] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:38:37,713] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:38:37,747] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:38:37,758] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.218607ms
[2022-09-10 21:38:37,950] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:38:37,983] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:38:37,995] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:38:37,996] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:38:38,009] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:38:38,109] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:38:38,915] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee757aa60dbc054884a2f671d955f54684job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:38:38,941] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee757aa60dbc054884a2f671d955f54684job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:38:38,987] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.338437ms
[2022-09-10 21:38:38,988] - [base_gsi:282] INFO - BUILD INDEX on default(employee757aa60dbc054884a2f671d955f54684job_title) USING GSI
[2022-09-10 21:38:40,018] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee757aa60dbc054884a2f671d955f54684job_title) USING GSI
[2022-09-10 21:38:40,044] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee757aa60dbc054884a2f671d955f54684job_title%29+USING+GSI
[2022-09-10 21:38:40,072] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.806072ms
[2022-09-10 21:38:41,103] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee757aa60dbc054884a2f671d955f54684job_title'
[2022-09-10 21:38:41,127] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee757aa60dbc054884a2f671d955f54684job_title%27
[2022-09-10 21:38:41,135] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.88728ms
[2022-09-10 21:38:42,166] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee757aa60dbc054884a2f671d955f54684job_title'
[2022-09-10 21:38:42,192] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee757aa60dbc054884a2f671d955f54684job_title%27
[2022-09-10 21:38:42,199] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.370884ms
[2022-09-10 21:38:42,580] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 21:38:42,840] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:38:43,567] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:38:44,201] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:38:44,234] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:38:44,259] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 21:38:44,263] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.300187ms
[2022-09-10 21:38:44,264] - [task:3246] INFO - {'requestID': '4b9c6e3d-d130-4243-9800-223488d857df', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee757aa60dbc054884a2f671d955f54684job_title', 'index_id': 'b4a6629919c4ebf5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.300187ms', 'executionTime': '2.225654ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 21:38:44,264] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:38:44,264] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:38:44,265] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:38:44,265] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:38:44,265] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 21:38:44,266] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:38:44,266] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:38:44,266] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:38:44,266] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:38:45,265] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:38:45,295] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:38:45,322] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 21:38:45,455] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 122.653401ms
[2022-09-10 21:38:45,455] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:38:45,455] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:38:46,239] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:38:46,239] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:38:47,272] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee757aa60dbc054884a2f671d955f54684job_title'
[2022-09-10 21:38:47,300] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee757aa60dbc054884a2f671d955f54684job_title%27
[2022-09-10 21:38:47,312] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.929962ms
[2022-09-10 21:38:47,340] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee757aa60dbc054884a2f671d955f54684job_title ON default USING GSI
[2022-09-10 21:38:47,370] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee757aa60dbc054884a2f671d955f54684job_title+ON+default+USING+GSI
[2022-09-10 21:38:47,426] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.476891ms
[2022-09-10 21:38:47,464] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee757aa60dbc054884a2f671d955f54684job_title'
[2022-09-10 21:38:47,490] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee757aa60dbc054884a2f671d955f54684job_title%27
[2022-09-10 21:38:47,498] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.495428ms
[2022-09-10 21:38:47,498] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'faf4b1a5-6e82-44b1-b5b9-ccd04a6b4e24', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.495428ms', 'executionTime': '5.40831ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:38:47,608] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:38:47,612] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:47,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:48,044] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:48,101] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:38:48,102] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:38:48,171] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:38:48,228] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:38:48,231] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:48,231] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:48,641] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:48,699] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:38:48,700] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:38:48,770] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:38:48,826] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:38:48,827] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:38:48,827] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:38:48,853] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:38:48,880] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:38:48,889] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.879118ms
[2022-09-10 21:38:48,916] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:38:48,942] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:38:48,988] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.7418ms
[2022-09-10 21:38:49,062] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:38:49,063] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.366474333732572, 'mem_free': 13062213632, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:38:49,063] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:38:49,067] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:49,067] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:49,449] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:49,454] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:49,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:50,035] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:50,042] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:50,042] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:50,753] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:50,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:50,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:51,453] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:55,525] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:38:55,672] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:38:56,018] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:38:56,047] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:38:56,047] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:38:56,103] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:38:56,159] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:38:56,217] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:38:56,218] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:38:56,301] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:38:56,302] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:38:56,329] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:38:56,460] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:38:56,460] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:38:56,488] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:38:56,515] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:38:56,515] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:38:56,543] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:38:56,569] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:38:56,569] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:38:56,596] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:38:56,623] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:38:56,624] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:38:56,650] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:38:56,651] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:38:56,651] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:38:56,651] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 78.155s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_4'}
[2022-09-10 21:38:56,750] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:56,751] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:57,169] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:57,200] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:38:57,284] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:38:57,285] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-10 21:38:57,285] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:38:57,777] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:38:57,804] - [task:165] INFO -  {'uptime': '340', 'memoryTotal': 15466930176, 'memoryFree': 13065220096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:38:57,830] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:38:57,830] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:38:57,831] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:38:57,858] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:38:57,893] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:38:57,894] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:38:57,923] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:38:57,924] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:38:57,924] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:38:57,925] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:38:57,972] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:38:57,975] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:57,975] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:58,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:58,337] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:38:58,423] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:38:58,425] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:38:58,457] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:38:58,458] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:38:58,484] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:38:58,570] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:38:58,598] - [task:165] INFO -  {'uptime': '339', 'memoryTotal': 15466930176, 'memoryFree': 13141180416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:38:58,624] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:38:58,654] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:38:58,655] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:38:58,707] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:38:58,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:58,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:59,105] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:59,107] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:38:59,191] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:38:59,192] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:38:59,224] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:38:59,225] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:38:59,250] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:38:59,338] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:38:59,367] - [task:165] INFO -  {'uptime': '336', 'memoryTotal': 15466930176, 'memoryFree': 13141282816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:38:59,394] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:38:59,422] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:38:59,423] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:38:59,474] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:38:59,478] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:38:59,479] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:38:59,877] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:38:59,878] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:38:59,967] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:38:59,968] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:38:59,999] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:39:00,000] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:39:00,028] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:39:00,116] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:39:00,143] - [task:165] INFO -  {'uptime': '335', 'memoryTotal': 15466930176, 'memoryFree': 13141557248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:39:00,169] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:39:00,198] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:39:00,198] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:39:00,250] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:39:00,253] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:39:00,253] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:39:00,660] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:39:00,661] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:39:00,750] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:39:00,752] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:39:00,784] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:39:00,785] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:39:00,812] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:39:00,869] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:39:01,406] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:39:06,412] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:39:06,495] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:39:06,502] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:39:06,502] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:39:06,894] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:39:06,895] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:39:06,985] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:39:06,986] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:39:06,986] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:39:07,995] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:39:08,049] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:39:08,050] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:39:08,947] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:39:09,253] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:39:09,531] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:39:09,536] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:39:09,600] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:39:09,600] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:39:09,979] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:39:09,983] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:39:09,983] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:39:10,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:39:10,382] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:39:10,382] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:39:10,969] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:39:10,979] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:39:10,979] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:39:11,628] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:39:15,969] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:39:15,969] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.41747191766816, 'mem_free': 13143920640, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:39:15,970] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:39:15,970] - [basetestcase:467] INFO - Time to execute basesetup : 19.22197151184082
[2022-09-10 21:39:16,021] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:39:16,021] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:39:16,074] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:39:16,074] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:39:16,127] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:39:16,127] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:39:16,183] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:39:16,184] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:39:16,238] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:39:16,294] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:39:16,295] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:39:16,295] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:39:21,304] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:39:21,307] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:39:21,307] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:39:21,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:39:22,707] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:39:22,952] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:39:25,451] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:39:25,467] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-10 21:39:26,149] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:39:26,149] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:39:26,149] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:39:56,177] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:39:56,209] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:39:56,237] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:39:56,306] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.590133ms
[2022-09-10 21:39:56,306] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'cf5d78df-2e39-4073-8d8d-e3abcb5b7bc2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.590133ms', 'executionTime': '66.531595ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:39:56,306] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:39:56,333] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:39:56,359] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:39:57,071] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 710.174543ms
[2022-09-10 21:39:57,072] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:39:57,199] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:39:57,238] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:39:57,251] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.248055ms
[2022-09-10 21:39:57,443] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:39:57,484] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:39:57,508] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:39:57,512] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:39:57,524] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:39:57,587] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:39:58,401] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeecbdbbb6cd1b64d018cf79335f5369b38job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:39:58,426] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeecbdbbb6cd1b64d018cf79335f5369b38job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:39:58,482] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.484473ms
[2022-09-10 21:39:58,482] - [base_gsi:282] INFO - BUILD INDEX on default(employeecbdbbb6cd1b64d018cf79335f5369b38job_title) USING GSI
[2022-09-10 21:39:59,513] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeecbdbbb6cd1b64d018cf79335f5369b38job_title) USING GSI
[2022-09-10 21:39:59,539] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeecbdbbb6cd1b64d018cf79335f5369b38job_title%29+USING+GSI
[2022-09-10 21:39:59,562] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.598824ms
[2022-09-10 21:40:00,595] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeecbdbbb6cd1b64d018cf79335f5369b38job_title'
[2022-09-10 21:40:00,621] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbdbbb6cd1b64d018cf79335f5369b38job_title%27
[2022-09-10 21:40:00,631] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.951122ms
[2022-09-10 21:40:01,662] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeecbdbbb6cd1b64d018cf79335f5369b38job_title'
[2022-09-10 21:40:01,688] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbdbbb6cd1b64d018cf79335f5369b38job_title%27
[2022-09-10 21:40:01,697] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.83376ms
[2022-09-10 21:40:02,171] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 21:40:02,343] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:40:03,112] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:40:03,298] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-10 21:40:03,298] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:40:03,476] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-10 21:40:03,477] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-10 21:40:03,477] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (1941029197, 0, b'')
[2022-09-10 21:40:03,700] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:40:03,728] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:40:03,753] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 21:40:03,757] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.105258ms
[2022-09-10 21:40:03,757] - [task:3246] INFO - {'requestID': '388fbfbc-7f96-4b38-8b2a-a120b3a888d7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeecbdbbb6cd1b64d018cf79335f5369b38job_title', 'index_id': 'edae9451195f8b07', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.105258ms', 'executionTime': '2.040035ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 21:40:03,758] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:40:03,758] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:40:03,758] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:40:03,759] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:40:03,759] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 21:40:03,759] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:40:03,759] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:40:03,759] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:40:03,760] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:40:04,759] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:40:04,788] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:40:04,814] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 21:40:04,931] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 109.373195ms
[2022-09-10 21:40:04,932] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:40:04,932] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:40:05,710] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:40:05,710] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:40:06,742] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeecbdbbb6cd1b64d018cf79335f5369b38job_title'
[2022-09-10 21:40:06,769] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbdbbb6cd1b64d018cf79335f5369b38job_title%27
[2022-09-10 21:40:06,779] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.917709ms
[2022-09-10 21:40:06,806] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeecbdbbb6cd1b64d018cf79335f5369b38job_title ON default USING GSI
[2022-09-10 21:40:06,833] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeecbdbbb6cd1b64d018cf79335f5369b38job_title+ON+default+USING+GSI
[2022-09-10 21:40:06,880] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.059304ms
[2022-09-10 21:40:06,912] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeecbdbbb6cd1b64d018cf79335f5369b38job_title'
[2022-09-10 21:40:06,942] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbdbbb6cd1b64d018cf79335f5369b38job_title%27
[2022-09-10 21:40:06,949] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.3804ms
[2022-09-10 21:40:06,949] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '65699a72-be88-4f75-ba66-ea99efb7dc23', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.3804ms', 'executionTime': '5.318964ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:40:07,059] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:40:07,063] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:07,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:07,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:07,605] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:40:07,605] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:40:07,681] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:40:07,741] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:40:07,744] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:07,744] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:08,225] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:08,285] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:40:08,286] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:40:08,364] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:40:08,420] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:40:08,420] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:40:08,420] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:40:08,445] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:40:08,471] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:40:08,481] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.033979ms
[2022-09-10 21:40:08,508] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:40:08,534] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:40:08,585] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.926964ms
[2022-09-10 21:40:08,655] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:40:08,655] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 9.277478161147089, 'mem_free': 13001363456, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:40:08,656] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:40:08,660] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:08,660] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:09,099] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:09,104] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:09,104] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:09,841] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:09,849] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:09,849] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:10,632] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:10,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:10,640] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:11,414] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:16,033] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:40:16,182] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:40:16,602] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:40:16,629] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:40:16,629] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:40:16,685] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:40:16,737] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:40:16,794] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:40:16,795] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:40:16,875] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:40:16,876] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:40:16,900] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:40:17,027] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:40:17,028] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:40:17,055] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:40:17,080] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:40:17,080] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:40:17,107] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:40:17,135] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:40:17,135] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:40:17,164] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:40:17,190] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:40:17,190] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:40:17,217] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:40:17,218] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:40:17,218] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:40:17,218] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.523s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_5'}
[2022-09-10 21:40:17,332] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:17,333] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:17,744] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:17,778] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:40:17,858] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:40:17,859] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-10 21:40:17,859] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:40:18,359] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:40:18,385] - [task:165] INFO -  {'uptime': '421', 'memoryTotal': 15466930176, 'memoryFree': 13016027136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:40:18,410] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:40:18,411] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:40:18,411] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:40:18,444] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:40:18,478] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:40:18,479] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:40:18,506] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:40:18,508] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:40:18,508] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:40:18,508] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:40:18,557] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:40:18,559] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:18,560] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:18,987] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:18,988] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:40:19,089] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:40:19,090] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:40:19,120] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:40:19,121] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:40:19,147] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:40:19,234] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:40:19,261] - [task:165] INFO -  {'uptime': '420', 'memoryTotal': 15466930176, 'memoryFree': 13109489664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:40:19,288] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:40:19,319] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:40:19,319] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:40:19,372] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:40:19,375] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:19,375] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:19,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:19,797] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:40:19,893] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:40:19,895] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:40:19,925] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:40:19,926] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:40:19,952] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:40:20,042] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:40:20,068] - [task:165] INFO -  {'uptime': '416', 'memoryTotal': 15466930176, 'memoryFree': 13108563968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:40:20,094] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:40:20,121] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:40:20,122] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:40:20,174] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:40:20,178] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:20,179] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:20,612] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:20,614] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:40:20,702] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:40:20,703] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:40:20,732] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:40:20,733] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:40:20,759] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:40:20,842] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:40:20,869] - [task:165] INFO -  {'uptime': '416', 'memoryTotal': 15466930176, 'memoryFree': 13109633024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:40:20,894] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:40:20,927] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:40:20,927] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:40:20,976] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:40:20,980] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:20,980] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:21,390] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:21,391] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:40:21,481] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:40:21,482] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:40:21,511] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:40:21,512] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:40:21,539] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:40:21,595] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:40:22,159] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:40:27,165] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:40:27,249] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:40:27,254] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:27,254] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:27,713] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:27,715] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:40:27,810] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:40:27,812] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:40:27,813] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:40:28,743] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:40:28,797] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:40:28,798] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:40:29,700] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:40:30,008] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:40:30,294] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:40:30,298] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:40:30,358] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:30,358] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:30,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:30,825] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:30,825] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:31,264] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:31,268] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:31,269] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:31,992] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:31,999] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:32,000] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:32,754] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:37,729] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:40:37,730] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.0202848338336, 'mem_free': 13116157952, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:40:37,730] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:40:37,730] - [basetestcase:467] INFO - Time to execute basesetup : 20.400255918502808
[2022-09-10 21:40:37,780] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:40:37,780] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:40:37,833] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:40:37,833] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:40:37,890] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:40:37,890] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:40:37,943] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:40:37,943] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:40:37,994] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:40:38,056] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:40:38,057] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:40:38,057] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:40:43,069] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:40:43,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:40:43,073] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:40:43,526] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:40:44,516] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:40:44,775] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:40:47,227] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:40:47,243] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 21:40:48,298] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:40:48,298] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:40:48,298] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:41:18,313] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:41:18,342] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:41:18,366] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:41:18,435] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.897086ms
[2022-09-10 21:41:18,436] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b31bc94a-ebe7-4027-983e-b069b71100d8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.897086ms', 'executionTime': '66.82459ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:41:18,436] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:41:18,461] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:41:18,485] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:41:19,215] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 727.334153ms
[2022-09-10 21:41:19,216] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:41:19,271] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:41:19,307] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:41:19,315] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.654101ms
[2022-09-10 21:41:19,524] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:41:19,560] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:41:19,575] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:41:19,575] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:41:19,591] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:41:19,668] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:41:20,485] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee76e923c73e904b88802d04f5aad974cejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:41:20,511] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee76e923c73e904b88802d04f5aad974cejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:41:20,569] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.548256ms
[2022-09-10 21:41:20,570] - [base_gsi:282] INFO - BUILD INDEX on default(employee76e923c73e904b88802d04f5aad974cejob_title) USING GSI
[2022-09-10 21:41:21,600] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee76e923c73e904b88802d04f5aad974cejob_title) USING GSI
[2022-09-10 21:41:21,625] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee76e923c73e904b88802d04f5aad974cejob_title%29+USING+GSI
[2022-09-10 21:41:21,650] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.716244ms
[2022-09-10 21:41:22,683] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee76e923c73e904b88802d04f5aad974cejob_title'
[2022-09-10 21:41:22,709] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee76e923c73e904b88802d04f5aad974cejob_title%27
[2022-09-10 21:41:22,719] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.561593ms
[2022-09-10 21:41:23,748] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee76e923c73e904b88802d04f5aad974cejob_title'
[2022-09-10 21:41:23,773] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee76e923c73e904b88802d04f5aad974cejob_title%27
[2022-09-10 21:41:23,783] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.652085ms
[2022-09-10 21:41:24,268] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:41:24,435] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:41:28,098] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:41:28,789] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:41:28,822] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:41:28,849] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 21:41:28,854] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.9763ms
[2022-09-10 21:41:28,854] - [task:3246] INFO - {'requestID': '542bc6d4-9161-4122-8116-42b822775f31', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee76e923c73e904b88802d04f5aad974cejob_title', 'index_id': '65129171e546a0a0', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.9763ms', 'executionTime': '2.88825ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 21:41:28,855] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:41:28,855] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:41:28,855] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:41:28,856] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:41:28,856] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 21:41:28,857] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:41:28,857] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:41:28,857] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:41:28,857] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:41:29,856] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:41:29,887] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:41:29,912] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 21:41:30,133] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 207.272312ms
[2022-09-10 21:41:30,134] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:41:30,135] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:41:31,671] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:41:31,671] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:41:32,703] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee76e923c73e904b88802d04f5aad974cejob_title'
[2022-09-10 21:41:32,730] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee76e923c73e904b88802d04f5aad974cejob_title%27
[2022-09-10 21:41:32,740] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.917018ms
[2022-09-10 21:41:32,770] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee76e923c73e904b88802d04f5aad974cejob_title ON default USING GSI
[2022-09-10 21:41:32,796] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee76e923c73e904b88802d04f5aad974cejob_title+ON+default+USING+GSI
[2022-09-10 21:41:32,845] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.352413ms
[2022-09-10 21:41:32,883] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee76e923c73e904b88802d04f5aad974cejob_title'
[2022-09-10 21:41:32,910] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee76e923c73e904b88802d04f5aad974cejob_title%27
[2022-09-10 21:41:32,917] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.470222ms
[2022-09-10 21:41:32,917] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '88ead787-1cab-498c-bc74-40572a476d4e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.470222ms', 'executionTime': '5.410752ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:41:33,035] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:41:33,038] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:33,039] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:33,606] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:33,661] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:41:33,662] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:41:33,751] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:41:33,808] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:41:33,811] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:33,811] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:34,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:34,436] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:41:34,437] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:41:34,526] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:41:34,582] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:41:34,582] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:41:34,583] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:41:34,609] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:41:34,635] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:41:34,645] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.823468ms
[2022-09-10 21:41:34,673] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:41:34,700] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:41:34,761] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 59.591681ms
[2022-09-10 21:41:34,832] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:41:34,832] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.32237291674291, 'mem_free': 12958998528, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:41:34,833] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:41:34,837] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:34,837] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:35,384] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:35,389] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:35,390] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:36,012] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:36,017] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:36,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:36,964] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:36,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:36,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:37,920] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:43,739] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:41:43,880] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:41:44,329] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:41:44,357] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:41:44,357] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:41:44,411] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:41:44,463] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:41:44,514] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:41:44,515] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:41:44,593] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:41:44,594] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:41:44,620] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:41:44,751] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:41:44,751] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:41:44,778] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:41:44,804] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:41:44,804] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:41:44,832] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:41:44,858] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:41:44,858] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:41:44,885] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:41:44,910] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:41:44,911] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:41:44,939] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:41:44,939] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:41:44,939] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:41:44,940] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 87.668s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_6'}
[2022-09-10 21:41:45,057] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:45,057] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:45,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:45,629] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:41:45,711] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:41:45,712] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-10 21:41:45,712] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:41:46,084] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:41:46,111] - [task:165] INFO -  {'uptime': '508', 'memoryTotal': 15466930176, 'memoryFree': 13033476096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:41:46,137] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:41:46,137] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:41:46,138] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:41:46,175] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:41:46,209] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:41:46,210] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:41:46,240] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:41:46,241] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:41:46,241] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:41:46,241] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:41:46,294] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:41:46,297] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:46,297] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:46,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:46,827] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:41:46,935] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:41:46,935] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:41:46,964] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:41:46,965] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:41:46,990] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:41:47,073] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:41:47,100] - [task:165] INFO -  {'uptime': '505', 'memoryTotal': 15466930176, 'memoryFree': 13033426944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:41:47,125] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:41:47,157] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:41:47,157] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:41:47,207] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:41:47,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:47,211] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:47,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:47,691] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:41:47,789] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:41:47,790] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:41:47,819] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:41:47,819] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:41:47,844] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:41:47,929] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:41:47,956] - [task:165] INFO -  {'uptime': '506', 'memoryTotal': 15466930176, 'memoryFree': 13032632320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:41:47,981] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:41:48,010] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:41:48,010] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:41:48,061] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:41:48,065] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:48,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:48,553] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:48,555] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:41:48,653] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:41:48,654] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:41:48,683] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:41:48,684] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:41:48,710] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:41:48,796] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:41:48,824] - [task:165] INFO -  {'uptime': '506', 'memoryTotal': 15466930176, 'memoryFree': 13028532224, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:41:48,852] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:41:48,882] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:41:48,882] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:41:48,938] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:41:48,941] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:48,941] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:49,485] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:49,486] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:41:49,593] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:41:49,594] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:41:49,624] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:41:49,624] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:41:49,651] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:41:49,712] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:41:50,258] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:41:55,263] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:41:55,348] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:41:55,352] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:55,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:55,869] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:55,870] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:41:55,973] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:41:55,974] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:41:55,975] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:41:56,832] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:41:56,891] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 21:41:56,892] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:41:57,797] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:41:58,109] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:41:58,398] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:41:58,401] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:41:58,463] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:58,463] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:58,979] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:58,984] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:58,984] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:41:59,586] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:41:59,597] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:41:59,597] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:42:00,500] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:42:00,509] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:42:00,509] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:42:01,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:42:07,157] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:42:07,157] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.98003374007344, 'mem_free': 13037727744, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:42:07,157] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:42:07,158] - [basetestcase:467] INFO - Time to execute basesetup : 22.102806329727173
[2022-09-10 21:42:07,208] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:42:07,208] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:42:07,262] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:42:07,262] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:42:07,319] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:42:07,320] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:42:07,373] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:42:07,373] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:42:07,424] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:42:07,486] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:42:07,487] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:42:07,487] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:42:12,497] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:42:12,500] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:42:12,501] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:42:13,049] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:42:14,071] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:42:14,343] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:42:16,557] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:42:16,574] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-10 21:42:17,943] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:42:17,943] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:42:17,943] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:42:47,951] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:42:47,979] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:42:48,005] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:42:48,074] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.31675ms
[2022-09-10 21:42:48,074] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9d4d6379-34a2-456c-be87-6324c2ed6dc0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.31675ms', 'executionTime': '67.242375ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:42:48,074] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:42:48,100] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:42:48,126] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:42:48,817] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 688.38582ms
[2022-09-10 21:42:48,817] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:42:48,883] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:42:48,924] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:42:48,932] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.108717ms
[2022-09-10 21:42:49,138] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:42:49,177] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:42:49,192] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:42:49,194] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:42:49,208] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:42:49,280] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:42:50,085] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:42:50,111] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:42:50,174] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.053122ms
[2022-09-10 21:42:50,175] - [base_gsi:282] INFO - BUILD INDEX on default(employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title) USING GSI
[2022-09-10 21:42:51,206] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title) USING GSI
[2022-09-10 21:42:51,232] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title%29+USING+GSI
[2022-09-10 21:42:51,252] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.516563ms
[2022-09-10 21:42:52,285] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title'
[2022-09-10 21:42:52,311] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title%27
[2022-09-10 21:42:52,320] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.758654ms
[2022-09-10 21:42:53,351] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title'
[2022-09-10 21:42:53,376] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title%27
[2022-09-10 21:42:53,386] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.550703ms
[2022-09-10 21:42:53,894] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-10 21:42:54,067] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:42:55,211] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:42:55,829] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-10 21:42:56,005] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:42:57,319] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:42:57,706] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:42:57,879] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:43:01,524] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:43:02,406] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:43:02,438] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:43:02,463] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-10 21:43:02,467] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.734412ms
[2022-09-10 21:43:02,468] - [task:3246] INFO - {'requestID': 'ccdf03e7-e50d-45ff-9e8e-d5ddd30fefba', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title', 'index_id': 'dc3bfb662aedcadd', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.734412ms', 'executionTime': '2.660692ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-10 21:43:02,468] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:43:02,468] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:43:02,469] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:43:02,469] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:43:02,469] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-10 21:43:02,469] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:43:02,470] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:43:02,470] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:43:02,470] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-10 21:43:03,469] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:43:03,498] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-10 21:43:03,524] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-10 21:43:03,898] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 146.018175ms
[2022-09-10 21:43:03,898] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:43:03,899] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:43:05,433] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:43:05,433] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-10 21:43:06,465] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title'
[2022-09-10 21:43:06,491] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title%27
[2022-09-10 21:43:06,501] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.006277ms
[2022-09-10 21:43:06,527] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title ON default USING GSI
[2022-09-10 21:43:06,555] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title+ON+default+USING+GSI
[2022-09-10 21:43:06,598] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.04149ms
[2022-09-10 21:43:06,643] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title'
[2022-09-10 21:43:06,672] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2fc1bf4c10ea45c595143d7d9aa9eb80job_title%27
[2022-09-10 21:43:06,682] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.794981ms
[2022-09-10 21:43:06,682] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9056cd80-a14c-4263-bcb1-a8c4aad6ca92', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.794981ms', 'executionTime': '6.713876ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:43:06,783] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:43:06,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:06,785] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:07,476] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:07,532] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:43:07,532] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:43:07,639] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:43:07,697] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:43:07,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:07,700] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:08,349] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:08,406] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:43:08,406] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:43:08,520] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:43:08,575] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:43:08,575] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:43:08,575] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:43:08,600] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:43:08,626] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:43:08,636] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.341305ms
[2022-09-10 21:43:08,663] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:43:08,692] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:43:08,743] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.08607ms
[2022-09-10 21:43:08,832] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:43:08,832] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.24932622169849, 'mem_free': 12746592256, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:43:08,832] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:43:08,837] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:08,837] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:09,501] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:09,506] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:09,506] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:10,207] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:10,212] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:10,212] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:11,092] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:11,099] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:11,100] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:12,272] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:19,864] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:43:20,005] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:43:20,395] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:43:20,423] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:43:20,423] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:43:20,478] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:43:20,532] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:43:20,584] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:43:20,585] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:43:20,663] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:43:20,664] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:43:20,689] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:43:20,818] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:43:20,818] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:43:20,844] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:43:20,869] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:43:20,869] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:43:20,895] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:43:20,921] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:43:20,921] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:43:20,947] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:43:20,973] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:43:20,973] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:43:20,998] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:43:20,998] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-10 21:43:20,999] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:43:20,999] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 95.996s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_7'}
[2022-09-10 21:43:21,144] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:21,145] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:21,752] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:21,785] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:43:21,867] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:43:21,867] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-10 21:43:21,868] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:43:22,171] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:43:22,199] - [task:165] INFO -  {'uptime': '604', 'memoryTotal': 15466930176, 'memoryFree': 12969996288, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:43:22,227] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:43:22,227] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:43:22,228] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:43:22,259] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:43:22,302] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:43:22,302] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:43:22,331] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:43:22,332] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:43:22,332] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:43:22,333] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:43:22,380] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:43:22,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:22,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:22,999] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:23,000] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:43:23,119] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:43:23,120] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:43:23,151] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:43:23,152] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:43:23,178] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:43:23,272] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:43:23,302] - [task:165] INFO -  {'uptime': '601', 'memoryTotal': 15466930176, 'memoryFree': 12971560960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:43:23,333] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:43:23,361] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:43:23,361] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:43:23,411] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:43:23,414] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:23,414] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:23,989] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:23,990] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:43:24,103] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:43:24,104] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:43:24,134] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:43:24,135] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:43:24,161] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:43:24,243] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:43:24,269] - [task:165] INFO -  {'uptime': '602', 'memoryTotal': 15466930176, 'memoryFree': 12972036096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:43:24,295] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:43:24,323] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:43:24,323] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:43:24,372] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:43:24,377] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:24,377] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:24,944] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:24,946] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:43:25,059] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:43:25,060] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:43:25,092] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:43:25,092] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:43:25,117] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:43:25,202] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:43:25,229] - [task:165] INFO -  {'uptime': '602', 'memoryTotal': 15466930176, 'memoryFree': 12971577344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:43:25,255] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:43:25,282] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:43:25,282] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:43:25,334] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:43:25,337] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:25,337] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:25,915] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:25,916] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:43:26,026] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:43:26,027] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:43:26,056] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:43:26,057] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:43:26,083] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:43:26,139] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:43:26,805] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:43:31,808] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:43:31,895] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:43:31,899] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:31,899] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:32,542] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:32,544] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:43:32,669] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:43:32,670] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:43:32,670] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:43:33,273] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:43:33,326] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:43:33,326] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:43:34,224] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:43:34,546] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:43:34,844] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:43:34,848] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 21:43:34,908] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:34,908] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:35,537] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:35,542] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:35,542] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:36,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:36,385] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:36,385] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:37,430] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:37,443] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:37,443] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:38,494] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:44,903] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:43:44,904] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.87032579313131, 'mem_free': 12983296000, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:43:44,904] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:43:44,904] - [basetestcase:467] INFO - Time to execute basesetup : 23.765074014663696
[2022-09-10 21:43:44,959] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:43:44,960] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:43:45,021] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:43:45,021] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:43:45,077] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:43:45,078] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:43:45,131] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:43:45,131] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:43:45,186] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:43:45,244] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:43:45,244] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:43:45,245] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:43:50,256] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:43:50,260] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:43:50,261] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:43:50,896] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:43:51,933] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:43:52,100] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:43:54,323] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:43:54,406] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:43:54,406] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:43:54,406] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:44:24,434] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:44:24,463] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:44:24,489] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:44:24,556] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.112845ms
[2022-09-10 21:44:24,556] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '275f1838-fe55-4d06-9daf-b8f6c0550f91', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.112845ms', 'executionTime': '65.044286ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:44:24,556] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:44:24,582] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:44:24,608] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:44:25,352] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 741.607194ms
[2022-09-10 21:44:25,352] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:44:25,406] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:44:25,457] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:44:25,465] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.107963ms
[2022-09-10 21:44:25,662] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:44:25,697] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:44:25,713] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:44:25,714] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:44:25,734] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:44:25,735] - [base_gsi:326] INFO - []
[2022-09-10 21:44:26,625] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed3435fb0c38d4ed9922ee7daedd81ad8job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:44:26,651] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed3435fb0c38d4ed9922ee7daedd81ad8job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:44:26,698] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.540547ms
[2022-09-10 21:44:26,736] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:44:26,762] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:44:26,834] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.200508ms
[2022-09-10 21:44:26,835] - [base_gsi:282] INFO - BUILD INDEX on default(employeed3435fb0c38d4ed9922ee7daedd81ad8job_title,employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr) USING GSI
[2022-09-10 21:44:27,866] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed3435fb0c38d4ed9922ee7daedd81ad8job_title,employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr) USING GSI
[2022-09-10 21:44:27,893] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed3435fb0c38d4ed9922ee7daedd81ad8job_title%2Cemployeed3435fb0c38d4ed9922ee7daedd81ad8join_yr%29+USING+GSI
[2022-09-10 21:44:27,934] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.394503ms
[2022-09-10 21:44:28,966] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed3435fb0c38d4ed9922ee7daedd81ad8job_title'
[2022-09-10 21:44:28,991] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed3435fb0c38d4ed9922ee7daedd81ad8job_title%27
[2022-09-10 21:44:28,999] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.403689ms
[2022-09-10 21:44:30,032] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed3435fb0c38d4ed9922ee7daedd81ad8job_title'
[2022-09-10 21:44:30,059] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed3435fb0c38d4ed9922ee7daedd81ad8job_title%27
[2022-09-10 21:44:30,069] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.623477ms
[2022-09-10 21:44:30,095] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr'
[2022-09-10 21:44:30,120] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr%27
[2022-09-10 21:44:30,123] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.591264ms
[2022-09-10 21:44:31,155] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed3435fb0c38d4ed9922ee7daedd81ad8job_title'
[2022-09-10 21:44:31,183] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed3435fb0c38d4ed9922ee7daedd81ad8job_title%27
[2022-09-10 21:44:31,192] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.247425ms
[2022-09-10 21:44:31,219] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed3435fb0c38d4ed9922ee7daedd81ad8job_title ON default USING GSI
[2022-09-10 21:44:31,247] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed3435fb0c38d4ed9922ee7daedd81ad8job_title+ON+default+USING+GSI
[2022-09-10 21:44:31,306] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.869543ms
[2022-09-10 21:44:31,343] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr'
[2022-09-10 21:44:31,371] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr%27
[2022-09-10 21:44:31,380] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.103447ms
[2022-09-10 21:44:31,407] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr ON default USING GSI
[2022-09-10 21:44:31,434] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr+ON+default+USING+GSI
[2022-09-10 21:44:31,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.256414ms
[2022-09-10 21:44:31,506] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed3435fb0c38d4ed9922ee7daedd81ad8job_title'
[2022-09-10 21:44:31,535] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed3435fb0c38d4ed9922ee7daedd81ad8job_title%27
[2022-09-10 21:44:31,542] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.852161ms
[2022-09-10 21:44:31,542] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a7ae8c72-7ada-450c-b191-03c2930b1052', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.852161ms', 'executionTime': '5.743227ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:44:31,570] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr'
[2022-09-10 21:44:31,596] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed3435fb0c38d4ed9922ee7daedd81ad8join_yr%27
[2022-09-10 21:44:31,599] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.436015ms
[2022-09-10 21:44:31,599] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '946a293e-4273-4caa-a6d3-005254fdfcbf', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.436015ms', 'executionTime': '1.368168ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:44:31,717] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:44:31,721] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:31,721] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:32,341] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:32,398] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:44:32,398] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:44:32,500] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:44:32,561] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:44:32,564] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:32,564] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:33,182] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:33,238] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:44:33,238] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:44:33,342] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:44:33,401] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:44:33,402] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:44:33,402] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:44:33,428] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:44:33,453] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:44:33,462] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.870754ms
[2022-09-10 21:44:33,488] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:44:33,514] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:44:33,567] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.659964ms
[2022-09-10 21:44:33,645] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:44:33,646] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.22536802887729, 'mem_free': 12861902848, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:44:33,646] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:44:33,651] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:33,651] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:34,319] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:34,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:34,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:34,960] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:34,965] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:34,965] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:35,994] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:36,002] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:36,002] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:37,057] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:43,622] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-10 21:44:43,772] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:44:44,837] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:44:44,865] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:44:44,865] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:44:44,931] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:44:44,988] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:44:45,046] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:44:45,047] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:44:45,132] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:44:45,134] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:44:45,160] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:44:45,292] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:44:45,293] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:44:45,320] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:44:45,346] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:44:45,346] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:44:45,372] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:44:45,397] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:44:45,397] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:44:45,424] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:44:45,449] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:44:45,449] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:44:45,475] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:44:45,475] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-10 21:44:45,475] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:44:45,476] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.389s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_8'}
[2022-09-10 21:44:45,555] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:45,555] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:46,150] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:46,184] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:44:46,268] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:44:46,268] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-10 21:44:46,269] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:44:46,582] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:44:46,610] - [task:165] INFO -  {'uptime': '689', 'memoryTotal': 15466930176, 'memoryFree': 12970852352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:44:46,638] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:44:46,638] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:44:46,638] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:44:46,676] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:44:46,713] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:44:46,717] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:44:46,746] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:44:46,747] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:44:46,747] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:44:46,747] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:44:46,795] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:44:46,798] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:46,799] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:47,411] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:47,413] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:44:47,527] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:44:47,528] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:44:47,558] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:44:47,559] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:44:47,584] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:44:47,669] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:44:47,696] - [task:165] INFO -  {'uptime': '686', 'memoryTotal': 15466930176, 'memoryFree': 12971134976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:44:47,721] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:44:47,749] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:44:47,749] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:44:47,798] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:44:47,802] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:47,802] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:48,394] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:48,395] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:44:48,518] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:44:48,519] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:44:48,550] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:44:48,551] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:44:48,576] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:44:48,663] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:44:48,695] - [task:165] INFO -  {'uptime': '687', 'memoryTotal': 15466930176, 'memoryFree': 12971204608, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:44:48,722] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:44:48,750] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:44:48,750] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:44:48,802] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:44:48,807] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:48,807] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:49,432] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:49,433] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:44:49,545] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:44:49,546] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:44:49,575] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:44:49,576] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:44:49,601] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:44:49,682] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:44:49,710] - [task:165] INFO -  {'uptime': '687', 'memoryTotal': 15466930176, 'memoryFree': 12971646976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:44:49,735] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:44:49,765] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:44:49,765] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:44:49,816] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:44:49,819] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:49,819] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:50,398] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:50,399] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:44:50,521] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:44:50,522] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:44:50,553] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:44:50,554] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:44:50,580] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:44:50,638] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:44:51,174] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:44:56,177] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:44:56,266] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:44:56,276] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:56,276] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:44:56,916] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:44:56,917] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:44:57,040] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:44:57,041] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:44:57,042] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:44:57,753] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:44:57,811] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 21:44:57,813] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:44:58,876] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:44:59,211] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:44:59,484] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:44:59,487] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 21:44:59,544] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:44:59,544] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:00,168] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:00,174] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:00,174] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:01,049] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:01,058] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:01,058] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:02,168] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:02,180] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:02,180] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:03,269] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:09,686] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:45:09,686] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.34899253609895, 'mem_free': 12968861696, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:45:09,687] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:45:09,687] - [basetestcase:467] INFO - Time to execute basesetup : 24.134130001068115
[2022-09-10 21:45:09,739] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:45:09,739] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:45:09,794] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:45:09,794] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:45:09,851] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:45:09,851] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:45:09,911] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:45:09,911] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:45:09,964] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:45:10,025] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:45:10,026] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:45:10,026] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:45:15,037] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:45:15,040] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:15,040] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:15,660] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:16,691] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:45:16,977] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:45:19,114] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:45:19,190] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:45:19,191] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:45:19,191] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:45:49,221] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:45:49,250] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:45:49,275] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:45:49,342] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.238942ms
[2022-09-10 21:45:49,342] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1660e57a-dd6f-45b6-b991-924204bd0dd0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.238942ms', 'executionTime': '65.176322ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:45:49,343] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:45:49,368] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:45:49,393] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:45:50,104] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 708.455175ms
[2022-09-10 21:45:50,104] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:45:50,166] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:45:50,202] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:45:50,211] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.205115ms
[2022-09-10 21:45:50,404] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:45:50,437] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:45:50,456] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:45:50,456] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:45:50,473] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:45:50,475] - [base_gsi:326] INFO - []
[2022-09-10 21:45:51,370] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee31f8030d71e84073814b600bbf06fa77job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:45:51,395] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee31f8030d71e84073814b600bbf06fa77job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:45:51,459] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.971515ms
[2022-09-10 21:45:51,460] - [base_gsi:282] INFO - BUILD INDEX on default(employee31f8030d71e84073814b600bbf06fa77job_title_join_yr) USING GSI
[2022-09-10 21:45:52,490] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee31f8030d71e84073814b600bbf06fa77job_title_join_yr) USING GSI
[2022-09-10 21:45:52,516] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee31f8030d71e84073814b600bbf06fa77job_title_join_yr%29+USING+GSI
[2022-09-10 21:45:52,537] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.952366ms
[2022-09-10 21:45:53,568] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee31f8030d71e84073814b600bbf06fa77job_title_join_yr'
[2022-09-10 21:45:53,593] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee31f8030d71e84073814b600bbf06fa77job_title_join_yr%27
[2022-09-10 21:45:53,602] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.307043ms
[2022-09-10 21:45:54,633] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee31f8030d71e84073814b600bbf06fa77job_title_join_yr'
[2022-09-10 21:45:54,661] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee31f8030d71e84073814b600bbf06fa77job_title_join_yr%27
[2022-09-10 21:45:54,671] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.727208ms
[2022-09-10 21:45:55,702] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee31f8030d71e84073814b600bbf06fa77job_title_join_yr'
[2022-09-10 21:45:55,729] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee31f8030d71e84073814b600bbf06fa77job_title_join_yr%27
[2022-09-10 21:45:55,739] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.038811ms
[2022-09-10 21:45:55,765] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee31f8030d71e84073814b600bbf06fa77job_title_join_yr ON default USING GSI
[2022-09-10 21:45:55,792] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee31f8030d71e84073814b600bbf06fa77job_title_join_yr+ON+default+USING+GSI
[2022-09-10 21:45:55,831] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.592856ms
[2022-09-10 21:45:55,879] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee31f8030d71e84073814b600bbf06fa77job_title_join_yr'
[2022-09-10 21:45:55,910] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee31f8030d71e84073814b600bbf06fa77job_title_join_yr%27
[2022-09-10 21:45:55,918] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.431591ms
[2022-09-10 21:45:55,918] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '932f2ce0-0a8b-4150-bb5f-78fe1f414428', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.431591ms', 'executionTime': '5.359018ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:45:56,025] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:45:56,028] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:56,028] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:56,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:56,746] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:45:56,747] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:45:56,853] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:45:56,916] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:45:56,919] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:56,919] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:57,559] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:57,614] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:45:57,614] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:45:57,714] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:45:57,770] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:45:57,770] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:45:57,770] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:45:57,795] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:45:57,821] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:45:57,832] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.894257ms
[2022-09-10 21:45:57,860] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:45:57,888] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:45:57,931] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.487288ms
[2022-09-10 21:45:58,003] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:45:58,003] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.248296433135313, 'mem_free': 12855996416, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:45:58,003] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:45:58,009] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:58,009] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:58,648] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:58,653] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:58,654] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:45:59,767] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:45:59,775] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:45:59,775] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:00,914] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:00,924] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:00,925] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:02,015] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:08,264] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-10 21:46:08,405] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:46:09,243] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:46:09,271] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:46:09,271] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:46:09,326] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:46:09,378] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:46:09,431] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:46:09,432] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:46:09,510] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:46:09,513] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:46:09,549] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:46:09,679] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:46:09,679] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:46:09,706] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:46:09,734] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:46:09,734] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:46:09,760] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:46:09,785] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:46:09,786] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:46:09,812] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:46:09,839] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:46:09,839] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:46:09,867] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:46:09,867] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-10 21:46:09,867] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:46:09,868] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.369s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_9'}
[2022-09-10 21:46:09,953] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:09,953] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:10,596] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:10,629] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:46:10,707] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:46:10,708] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-10 21:46:10,708] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:46:10,979] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:46:11,005] - [task:165] INFO -  {'uptime': '773', 'memoryTotal': 15466930176, 'memoryFree': 12970631168, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:46:11,031] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:46:11,031] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:46:11,032] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:46:11,067] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:46:11,103] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:46:11,103] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:46:11,131] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:46:11,132] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:46:11,132] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:46:11,133] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:46:11,181] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:46:11,184] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:11,184] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:11,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:11,797] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:46:11,924] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:46:11,925] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:46:11,956] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:46:11,957] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:46:11,983] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:46:12,071] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:46:12,097] - [task:165] INFO -  {'uptime': '771', 'memoryTotal': 15466930176, 'memoryFree': 12972355584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:46:12,123] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:46:12,151] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:46:12,151] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:46:12,203] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:46:12,206] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:12,207] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:12,842] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:12,843] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:46:12,959] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:46:12,960] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:46:12,988] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:46:12,989] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:46:13,018] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:46:13,100] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:46:13,127] - [task:165] INFO -  {'uptime': '772', 'memoryTotal': 15466930176, 'memoryFree': 12972072960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:46:13,152] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:46:13,180] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:46:13,180] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:46:13,230] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:46:13,235] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:13,235] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:13,843] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:13,844] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:46:13,963] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:46:13,964] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:46:13,993] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:46:13,994] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:46:14,023] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:46:14,110] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:46:14,136] - [task:165] INFO -  {'uptime': '772', 'memoryTotal': 15466930176, 'memoryFree': 12959563776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:46:14,162] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:46:14,190] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:46:14,191] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:46:14,243] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:46:14,248] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:14,248] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:14,888] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:14,889] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:46:15,016] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:46:15,017] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:46:15,049] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:46:15,049] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:46:15,075] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:46:15,131] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:46:15,658] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:46:20,663] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:46:20,747] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:46:20,761] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:20,762] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:21,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:21,421] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:46:21,547] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:46:21,548] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:46:21,548] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:46:22,244] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:46:22,298] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:46:22,298] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:46:23,240] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:46:23,556] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:46:23,853] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:46:23,857] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 21:46:23,919] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:23,919] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:24,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:24,599] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:24,600] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:25,582] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:25,591] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:25,591] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:26,742] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:26,751] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:26,751] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:27,839] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:34,608] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:46:34,609] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.32424271018212, 'mem_free': 12968992768, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:46:34,609] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:46:34,609] - [basetestcase:467] INFO - Time to execute basesetup : 24.658825874328613
[2022-09-10 21:46:34,660] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:46:34,660] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:46:34,716] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:46:34,716] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:46:34,770] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:46:34,771] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:46:34,823] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:46:34,824] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:46:34,875] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:46:34,933] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:46:34,933] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:46:34,933] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:46:39,946] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:46:39,949] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:46:39,949] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:46:40,543] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:46:41,681] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:46:41,846] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:46:44,013] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:46:44,090] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:46:44,091] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:46:44,091] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:47:14,121] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:47:14,156] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:47:14,183] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:47:14,252] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.076577ms
[2022-09-10 21:47:14,252] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0d31999d-8df8-4679-9ccf-b00575e67c7d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.076577ms', 'executionTime': '67.014516ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:47:14,252] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:47:14,281] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:47:14,306] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:47:15,021] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 713.704299ms
[2022-09-10 21:47:15,022] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:47:15,120] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:47:15,159] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:47:15,170] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.538072ms
[2022-09-10 21:47:15,381] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:47:15,418] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:47:15,440] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:47:15,440] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:47:15,452] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:47:15,516] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:47:16,335] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee13d12866cbb249cab973cc474bf0de27join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:47:16,362] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee13d12866cbb249cab973cc474bf0de27join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:47:16,433] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.764774ms
[2022-09-10 21:47:16,433] - [base_gsi:282] INFO - BUILD INDEX on default(`employee13d12866cbb249cab973cc474bf0de27join_yr`) USING GSI
[2022-09-10 21:47:17,462] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee13d12866cbb249cab973cc474bf0de27join_yr`) USING GSI
[2022-09-10 21:47:17,488] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee13d12866cbb249cab973cc474bf0de27join_yr%60%29+USING+GSI
[2022-09-10 21:47:17,507] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.433029ms
[2022-09-10 21:47:17,548] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee13d12866cbb249cab973cc474bf0de27join_yr'
[2022-09-10 21:47:17,576] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee13d12866cbb249cab973cc474bf0de27join_yr%27
[2022-09-10 21:47:17,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.956391ms
[2022-09-10 21:47:18,617] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee13d12866cbb249cab973cc474bf0de27join_yr'
[2022-09-10 21:47:18,643] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee13d12866cbb249cab973cc474bf0de27join_yr%27
[2022-09-10 21:47:18,652] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.002722ms
[2022-09-10 21:47:19,683] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee13d12866cbb249cab973cc474bf0de27join_yr'
[2022-09-10 21:47:19,709] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee13d12866cbb249cab973cc474bf0de27join_yr%27
[2022-09-10 21:47:19,719] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.701729ms
[2022-09-10 21:47:20,750] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:47:20,776] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 21:47:20,780] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.7646ms
[2022-09-10 21:47:20,780] - [base_gsi:504] INFO - {'requestID': '13c543a1-b0b5-40fa-8830-f11aa0c77a49', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee13d12866cbb249cab973cc474bf0de27join_yr', 'index_id': '3d655873048209a9', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.7646ms', 'executionTime': '2.686883ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 21:47:20,781] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:47:20,781] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:47:20,782] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:47:20,782] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:47:20,783] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:47:20,783] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:47:20,784] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:47:20,784] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:47:20,844] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:47:20,882] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:47:20,914] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 21:47:21,058] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 130.107112ms
[2022-09-10 21:47:21,058] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:47:21,059] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:47:22,957] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:47:22,982] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee13d12866cbb249cab973cc474bf0de27join_yr'
[2022-09-10 21:47:23,012] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee13d12866cbb249cab973cc474bf0de27join_yr%27
[2022-09-10 21:47:23,019] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.610312ms
[2022-09-10 21:47:23,019] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c4623ff0-f1b3-4dea-80b2-dc490d878366', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.610312ms', 'executionTime': '4.543562ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:47:23,120] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:47:23,123] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:23,123] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:23,851] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:23,909] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:47:23,910] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:47:24,022] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:47:24,081] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:47:24,084] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:24,084] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:24,746] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:24,802] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:47:24,803] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:47:24,916] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:47:24,971] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:47:24,971] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:47:24,971] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:47:24,998] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:47:25,026] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:47:25,034] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.298089ms
[2022-09-10 21:47:25,034] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4887a732-2fa0-435e-bf10-59643520a755', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.298089ms', 'executionTime': '6.236428ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:47:25,089] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:47:25,089] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.66081858830099, 'mem_free': 12836315136, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:47:25,089] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:47:25,093] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:25,093] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:25,781] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:25,786] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:25,786] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:26,484] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:26,489] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:26,490] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:27,451] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:27,460] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:27,460] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:28,593] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:35,958] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-10 21:47:36,097] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:47:36,152] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:47:36,208] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:47:36,262] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:47:36,262] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:47:36,340] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:47:36,341] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:47:36,366] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:47:36,496] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:47:36,497] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:47:36,524] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:47:36,550] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:47:36,550] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:47:36,577] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:47:36,603] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:47:36,603] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:47:36,630] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:47:36,655] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:47:36,655] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:47:36,682] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:47:36,682] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-10 21:47:36,682] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:47:36,683] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.784s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_10'}
[2022-09-10 21:47:36,761] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:36,762] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:37,460] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:37,493] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:47:37,576] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:47:37,576] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-10 21:47:37,577] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:47:37,788] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:47:37,822] - [task:165] INFO -  {'uptime': '857', 'memoryTotal': 15466930176, 'memoryFree': 12977115136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:47:37,848] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:47:37,848] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:47:37,849] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:47:37,885] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:47:37,923] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:47:37,924] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:47:37,954] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:47:37,955] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:47:37,955] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:47:37,956] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:47:38,005] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:47:38,009] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:38,009] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:38,630] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:38,631] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:47:38,751] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:47:38,752] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:47:38,782] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:47:38,782] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:47:38,807] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:47:38,891] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:47:38,918] - [task:165] INFO -  {'uptime': '857', 'memoryTotal': 15466930176, 'memoryFree': 12975300608, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:47:38,943] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:47:38,971] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:47:38,971] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:47:39,024] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:47:39,029] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:39,029] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:39,709] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:39,710] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:47:39,842] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:47:39,843] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:47:39,875] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:47:39,876] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:47:39,904] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:47:39,996] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:47:40,023] - [task:165] INFO -  {'uptime': '858', 'memoryTotal': 15466930176, 'memoryFree': 12975927296, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:47:40,049] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:47:40,078] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:47:40,079] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:47:40,129] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:47:40,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:40,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:40,789] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:40,790] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:47:40,922] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:47:40,923] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:47:40,953] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:47:40,954] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:47:40,980] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:47:41,067] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:47:41,094] - [task:165] INFO -  {'uptime': '858', 'memoryTotal': 15466930176, 'memoryFree': 12975681536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:47:41,122] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:47:41,150] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:47:41,151] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:47:41,203] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:47:41,206] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:41,206] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:41,856] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:41,857] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:47:41,977] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:47:41,978] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:47:42,007] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:47:42,008] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:47:42,034] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:47:42,090] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:47:42,609] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:47:47,615] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:47:47,697] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:47:47,709] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:47,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:48,352] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:48,353] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:47:48,472] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:47:48,473] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:47:48,473] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:47:49,209] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:47:49,262] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:47:49,262] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:47:50,180] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:47:50,489] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:47:50,769] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:47:50,772] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 21:47:50,832] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:50,833] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:51,480] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:51,485] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:51,485] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:52,390] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:52,398] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:52,398] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:53,563] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:47:53,575] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:47:53,575] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:47:54,738] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:48:01,700] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:48:01,700] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.30866797886812, 'mem_free': 12971622400, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:48:01,700] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:48:01,701] - [basetestcase:467] INFO - Time to execute basesetup : 24.941790103912354
[2022-09-10 21:48:01,755] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:48:01,755] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:48:01,810] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:48:01,811] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:48:01,868] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:48:01,868] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:48:01,926] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:48:01,926] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:48:01,981] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:48:02,048] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:48:02,050] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:48:02,051] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:48:07,060] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:48:07,063] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:48:07,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:48:07,731] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:48:08,754] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:48:09,041] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:48:11,216] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:48:11,295] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:48:11,296] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:48:11,296] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:48:41,325] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:48:41,355] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:48:41,382] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:48:41,451] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.82956ms
[2022-09-10 21:48:41,451] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '871db533-5abe-4fd1-9608-3b4d6179774b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.82956ms', 'executionTime': '66.762951ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:48:41,451] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:48:41,477] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:48:41,501] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:48:42,228] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 724.834155ms
[2022-09-10 21:48:42,228] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:48:42,281] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:48:42,336] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:48:42,344] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.32362ms
[2022-09-10 21:48:42,543] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:48:42,579] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:48:42,595] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:48:42,596] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:48:42,624] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:48:42,688] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:48:43,502] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee95fbd50b35c948718854b99dad48a295join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:48:43,527] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee95fbd50b35c948718854b99dad48a295join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:48:43,584] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.976251ms
[2022-09-10 21:48:43,584] - [base_gsi:282] INFO - BUILD INDEX on default(`employee95fbd50b35c948718854b99dad48a295join_yr`) USING GSI
[2022-09-10 21:48:44,616] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee95fbd50b35c948718854b99dad48a295join_yr`) USING GSI
[2022-09-10 21:48:44,645] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee95fbd50b35c948718854b99dad48a295join_yr%60%29+USING+GSI
[2022-09-10 21:48:44,675] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 27.136044ms
[2022-09-10 21:48:44,721] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee95fbd50b35c948718854b99dad48a295join_yr'
[2022-09-10 21:48:44,750] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee95fbd50b35c948718854b99dad48a295join_yr%27
[2022-09-10 21:48:44,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.16395ms
[2022-09-10 21:48:45,790] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee95fbd50b35c948718854b99dad48a295join_yr'
[2022-09-10 21:48:45,817] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee95fbd50b35c948718854b99dad48a295join_yr%27
[2022-09-10 21:48:45,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.392079ms
[2022-09-10 21:48:46,858] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee95fbd50b35c948718854b99dad48a295join_yr'
[2022-09-10 21:48:46,883] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee95fbd50b35c948718854b99dad48a295join_yr%27
[2022-09-10 21:48:46,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.277941ms
[2022-09-10 21:48:47,920] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:48:47,946] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 21:48:47,950] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.932524ms
[2022-09-10 21:48:47,950] - [base_gsi:504] INFO - {'requestID': 'c63aea9f-4fbb-4e50-8688-a3fbe7b7a3e5', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee95fbd50b35c948718854b99dad48a295join_yr', 'index_id': 'f3ff7005e201e5e3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.932524ms', 'executionTime': '2.873137ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 21:48:47,951] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:48:47,951] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:48:47,952] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:48:47,952] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:48:47,952] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:48:47,952] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:48:47,953] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:48:47,953] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:48:48,011] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:48:48,049] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:48:48,077] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 21:48:48,212] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 121.611ms
[2022-09-10 21:48:48,212] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:48:48,213] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:48:49,382] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-10 21:48:49,437] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-10 21:48:49,466] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:48:49,492] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 21:48:49,496] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.85547ms
[2022-09-10 21:48:49,496] - [base_gsi:504] INFO - {'requestID': 'c6185bc8-6e2f-4d85-9191-5fe9a4a4e87e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee95fbd50b35c948718854b99dad48a295join_yr', 'index_id': 'f3ff7005e201e5e3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.85547ms', 'executionTime': '1.776846ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-10 21:48:49,496] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:48:49,497] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:48:49,497] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:48:49,497] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:48:49,497] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:48:49,498] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:48:49,498] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:48:49,498] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:48:49,550] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:48:49,589] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:48:49,614] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 21:48:49,755] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 132.509889ms
[2022-09-10 21:48:49,755] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:48:49,756] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:48:50,923] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee95fbd50b35c948718854b99dad48a295join_yr ON default USING GSI
[2022-09-10 21:48:50,954] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee95fbd50b35c948718854b99dad48a295join_yr+ON+default+USING+GSI
[2022-09-10 21:48:51,000] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.086262ms
[2022-09-10 21:48:51,035] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee95fbd50b35c948718854b99dad48a295join_yr'
[2022-09-10 21:48:51,061] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee95fbd50b35c948718854b99dad48a295join_yr%27
[2022-09-10 21:48:51,068] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.805528ms
[2022-09-10 21:48:51,068] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '71aa0e55-302f-48b8-be61-0e616948ec0c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.805528ms', 'executionTime': '5.733262ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:48:51,179] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:48:51,182] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:48:51,182] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:48:51,841] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:48:51,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:48:51,895] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:48:52,003] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:48:52,060] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:48:52,063] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:48:52,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:48:52,715] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:48:52,772] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:48:52,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:48:52,885] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:48:52,939] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:48:52,939] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:48:52,940] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:48:52,965] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:48:52,990] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:48:53,001] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.627875ms
[2022-09-10 21:48:53,028] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:48:53,053] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:48:53,103] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.690215ms
[2022-09-10 21:48:53,172] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:48:53,172] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.49400349314577, 'mem_free': 12798337024, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:48:53,172] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:48:53,175] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:48:53,176] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:48:53,844] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:48:53,848] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:48:53,848] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:48:55,010] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:48:55,018] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:48:55,018] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:48:56,181] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:48:56,193] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:48:56,193] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:48:57,397] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:04,285] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-10 21:49:04,427] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:49:04,834] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:49:04,863] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:49:04,864] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:49:04,920] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:49:04,972] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:49:05,028] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:49:05,029] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:49:05,110] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:49:05,111] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:49:05,136] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:49:05,265] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:49:05,266] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:49:05,292] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:49:05,318] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:49:05,318] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:49:05,344] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:49:05,369] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:49:05,369] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:49:05,396] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:49:05,421] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:49:05,421] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:49:05,448] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:49:05,448] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-10 21:49:05,448] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:49:05,449] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 88.742s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_11'}
[2022-09-10 21:49:05,528] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:05,528] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:06,239] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:06,273] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:49:06,360] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:49:06,360] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-10 21:49:06,361] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:49:06,556] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:49:06,585] - [task:165] INFO -  {'uptime': '949', 'memoryTotal': 15466930176, 'memoryFree': 12958990336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:49:06,613] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:49:06,613] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:49:06,613] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:49:06,644] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:49:06,686] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:49:06,687] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:49:06,716] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:49:06,717] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:49:06,717] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:49:06,717] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:49:06,766] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:49:06,769] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:06,769] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:07,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:07,457] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:49:07,582] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:49:07,583] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:49:07,612] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:49:07,612] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:49:07,638] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:49:07,722] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:49:07,749] - [task:165] INFO -  {'uptime': '947', 'memoryTotal': 15466930176, 'memoryFree': 12957360128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:49:07,773] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:49:07,801] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:49:07,801] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:49:07,851] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:49:07,854] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:07,854] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:08,518] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:08,520] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:49:08,651] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:49:08,652] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:49:08,682] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:49:08,683] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:49:08,709] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:49:08,796] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:49:08,824] - [task:165] INFO -  {'uptime': '948', 'memoryTotal': 15466930176, 'memoryFree': 12958273536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:49:08,850] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:49:08,879] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:49:08,879] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:49:08,931] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:49:08,935] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:08,936] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:09,606] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:09,607] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:49:09,738] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:49:09,739] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:49:09,769] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:49:09,770] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:49:09,795] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:49:09,885] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:49:09,912] - [task:165] INFO -  {'uptime': '948', 'memoryTotal': 15466930176, 'memoryFree': 12957884416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:49:09,939] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:49:09,966] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:49:09,966] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:49:10,018] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:49:10,020] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:10,020] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:10,730] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:10,732] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:49:10,853] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:49:10,854] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:49:10,883] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:49:10,884] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:49:10,910] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:49:10,966] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:49:11,607] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:49:16,611] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:49:16,696] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:49:16,701] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:16,702] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:17,372] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:17,374] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:49:17,501] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:49:17,503] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:49:17,503] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:49:18,091] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:49:18,146] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:49:18,146] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:49:19,046] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:49:19,351] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:49:19,605] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:49:19,609] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 21:49:19,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:19,663] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:20,335] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:20,340] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:20,340] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:21,348] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:21,356] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:21,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:22,554] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:22,563] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:22,563] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:23,772] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:30,711] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:49:30,712] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.84944481358642, 'mem_free': 12954886144, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:49:30,712] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:49:30,712] - [basetestcase:467] INFO - Time to execute basesetup : 25.186317920684814
[2022-09-10 21:49:30,765] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:49:30,765] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:49:30,819] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:49:30,819] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:49:30,872] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:49:30,873] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:49:30,927] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:49:30,927] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:49:30,980] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:49:31,041] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:49:31,042] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:49:31,042] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:49:36,054] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:49:36,058] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:49:36,058] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:49:36,756] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:49:37,944] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:49:38,120] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:49:41,212] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:49:41,287] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:49:41,288] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:49:41,288] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:50:11,311] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:50:11,342] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:50:11,368] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:50:11,439] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.577228ms
[2022-09-10 21:50:11,439] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a1673324-e832-4bd1-81d3-3181afc175b6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.577228ms', 'executionTime': '68.519213ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:50:11,439] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:50:11,466] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:50:11,491] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:50:12,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 737.46865ms
[2022-09-10 21:50:12,231] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:50:12,303] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:50:12,334] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:50:12,342] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.342038ms
[2022-09-10 21:50:12,544] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:50:12,579] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:50:12,596] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:50:12,597] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:50:12,615] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:50:12,679] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:50:13,501] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed56f3ab1a9604bf99bd8a9b716807706join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-10 21:50:13,527] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed56f3ab1a9604bf99bd8a9b716807706join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-10 21:50:13,598] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.890251ms
[2022-09-10 21:50:13,599] - [base_gsi:282] INFO - BUILD INDEX on default(`employeed56f3ab1a9604bf99bd8a9b716807706join_yr`) USING GSI
[2022-09-10 21:50:14,630] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeed56f3ab1a9604bf99bd8a9b716807706join_yr`) USING GSI
[2022-09-10 21:50:14,657] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeed56f3ab1a9604bf99bd8a9b716807706join_yr%60%29+USING+GSI
[2022-09-10 21:50:14,681] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.048624ms
[2022-09-10 21:50:14,731] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed56f3ab1a9604bf99bd8a9b716807706join_yr'
[2022-09-10 21:50:14,762] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed56f3ab1a9604bf99bd8a9b716807706join_yr%27
[2022-09-10 21:50:14,777] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 11.534642ms
[2022-09-10 21:50:15,808] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed56f3ab1a9604bf99bd8a9b716807706join_yr'
[2022-09-10 21:50:15,835] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed56f3ab1a9604bf99bd8a9b716807706join_yr%27
[2022-09-10 21:50:15,844] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.743008ms
[2022-09-10 21:50:16,877] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed56f3ab1a9604bf99bd8a9b716807706join_yr'
[2022-09-10 21:50:16,906] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed56f3ab1a9604bf99bd8a9b716807706join_yr%27
[2022-09-10 21:50:16,914] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.007617ms
[2022-09-10 21:50:17,945] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:50:17,971] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-10 21:50:17,975] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.522555ms
[2022-09-10 21:50:17,975] - [base_gsi:504] INFO - {'requestID': '4114f81e-a030-4c67-a48e-97d609850f2d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeed56f3ab1a9604bf99bd8a9b716807706join_yr', 'index_id': '3868fb05f296e23', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.522555ms', 'executionTime': '2.452052ms', 'resultCount': 1, 'resultSize': 925, 'serviceLoad': 6}}
[2022-09-10 21:50:17,976] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-10 21:50:17,976] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:50:17,976] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-10 21:50:17,976] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-10 21:50:17,977] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:50:17,977] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:50:17,977] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:50:17,977] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-10 21:50:18,037] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-10 21:50:18,075] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-10 21:50:18,100] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-10 21:50:18,280] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 166.616908ms
[2022-09-10 21:50:18,280] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-10 21:50:18,281] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-10 21:50:20,628] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 21:50:22,951] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:50:23,011] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-10 21:50:23,011] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:50:23,900] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:50:24,215] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:50:24,492] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:50:24,495] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-10 21:50:26,528] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:50:26,528] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:26,555] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:50:26,583] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:50:26,584] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:26,610] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:50:26,637] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:50:26,638] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:26,663] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:50:26,691] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:50:26,691] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:26,718] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:50:26,769] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:50:26,795] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed56f3ab1a9604bf99bd8a9b716807706join_yr'
[2022-09-10 21:50:26,820] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed56f3ab1a9604bf99bd8a9b716807706join_yr%27
[2022-09-10 21:50:26,864] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.198378ms
[2022-09-10 21:50:26,864] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '63117c78-bce6-496f-a66c-c9e60085b515', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '41.198378ms', 'executionTime': '41.139181ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:50:26,891] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed56f3ab1a9604bf99bd8a9b716807706join_yr'
[2022-09-10 21:50:26,918] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed56f3ab1a9604bf99bd8a9b716807706join_yr%27
[2022-09-10 21:50:26,922] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.31762ms
[2022-09-10 21:50:26,922] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fbb174e2-953d-46b1-9df5-2b7475370555', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.31762ms', 'executionTime': '1.258972ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:50:26,976] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:50:27,112] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:50:27,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:27,115] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:27,866] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:27,925] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:50:27,926] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:50:28,045] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:50:28,102] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:50:28,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:28,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:28,832] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:28,888] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:50:28,888] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:50:29,008] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:50:29,066] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:50:29,067] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:50:29,067] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:50:29,092] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:50:29,118] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:50:29,127] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.56165ms
[2022-09-10 21:50:29,127] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '411eac4c-ac82-42b6-a8f1-5763336e9344', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.56165ms', 'executionTime': '7.500081ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:50:29,127] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:50:29,154] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:50:29,179] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:50:29,182] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.24035ms
[2022-09-10 21:50:29,182] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bd4ab14c-217c-4e4f-9940-c688dda6ae1e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.24035ms', 'executionTime': '1.173866ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:50:29,232] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:50:29,232] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.6754183587717, 'mem_free': 12788203520, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:50:29,233] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:50:29,236] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:29,236] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:29,964] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:29,969] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:29,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:30,822] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:30,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:30,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:32,093] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:32,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:32,101] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:33,355] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:40,795] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 21:50:40,938] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:50:41,263] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:50:41,291] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:50:41,291] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:50:41,348] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:50:41,400] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:50:41,454] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:50:41,454] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:50:41,536] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:50:41,537] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:50:41,563] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:41,691] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:50:41,691] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:41,717] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:50:41,744] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:50:41,744] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:41,771] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:50:41,796] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:50:41,796] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:41,823] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:50:41,850] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:50:41,850] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:41,877] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:50:41,877] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-10 21:50:41,878] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:50:41,879] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.409s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-10_21-33-12/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-10 21:50:41,941] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:41,942] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:42,692] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:42,726] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:50:42,811] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-10 21:50:42,811] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-10 21:50:42,812] - [basetestcase:224] INFO - initializing cluster
[2022-09-10 21:50:42,969] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-10 21:50:42,998] - [task:165] INFO -  {'uptime': '1045', 'memoryTotal': 15466930176, 'memoryFree': 12945158144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:50:43,029] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-10 21:50:43,029] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-10 21:50:43,029] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-10 21:50:43,063] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-10 21:50:43,101] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-10 21:50:43,102] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-10 21:50:43,129] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-10 21:50:43,130] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-10 21:50:43,130] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-10 21:50:43,131] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-10 21:50:43,179] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:50:43,181] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:43,181] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:43,872] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:43,873] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:50:44,012] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:50:44,013] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:50:44,046] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:50:44,046] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:50:44,072] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:50:44,155] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-10 21:50:44,180] - [task:165] INFO -  {'uptime': '1042', 'memoryTotal': 15466930176, 'memoryFree': 12945760256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:50:44,205] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:50:44,233] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-10 21:50:44,233] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-10 21:50:44,282] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:50:44,285] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:44,285] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:44,960] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:44,961] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:50:45,095] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:50:45,096] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:50:45,125] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:50:45,126] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:50:45,151] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:50:45,238] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-10 21:50:45,263] - [task:165] INFO -  {'uptime': '1044', 'memoryTotal': 15466930176, 'memoryFree': 12936994816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:50:45,289] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:50:45,321] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-10 21:50:45,322] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-10 21:50:45,371] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:50:45,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:45,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:46,072] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:46,073] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:50:46,212] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:50:46,213] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:50:46,245] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:50:46,246] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:50:46,276] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:50:46,366] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-10 21:50:46,392] - [task:165] INFO -  {'uptime': '1043', 'memoryTotal': 15466930176, 'memoryFree': 12943982592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-10 21:50:46,418] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-10 21:50:46,447] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-10 21:50:46,447] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-10 21:50:46,500] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-10 21:50:46,503] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:46,503] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:47,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:47,242] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:50:47,378] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:50:47,380] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:50:47,411] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-10 21:50:47,412] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-10 21:50:47,438] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-10 21:50:47,495] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-10 21:50:48,134] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:50:53,139] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-10 21:50:53,230] - [basetestcase:262] INFO - done initializing cluster
[2022-09-10 21:50:53,233] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:53,234] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:53,951] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:53,952] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-10 21:50:54,085] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-10 21:50:54,086] - [remote_util:5249] INFO - b'ok'
[2022-09-10 21:50:54,087] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-10 21:50:54,642] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-10 21:50:54,695] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-10 21:50:54,696] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-10 21:50:55,603] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:50:55,932] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:50:56,238] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-10 21:50:56,241] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-10 21:50:56,307] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:56,307] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:57,061] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:57,065] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:57,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:58,230] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:58,238] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:58,238] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:50:59,487] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:50:59,496] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:50:59,496] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:00,777] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:51:08,152] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:51:08,152] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.75465509530843, 'mem_free': 12943847424, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:51:08,153] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:51:08,153] - [basetestcase:467] INFO - Time to execute basesetup : 26.214439153671265
[2022-09-10 21:51:08,206] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:51:08,206] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:51:08,261] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:51:08,261] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:51:08,313] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:51:08,314] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:51:08,367] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-10 21:51:08,367] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-10 21:51:08,421] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:51:08,485] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-10 21:51:08,485] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-10 21:51:08,487] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-10 21:51:13,496] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-10 21:51:13,499] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:51:13,499] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:14,219] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:51:15,399] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-10 21:51:15,571] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-10 21:51:18,526] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-10 21:51:18,601] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:51:18,601] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-10 21:51:18,601] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-10 21:51:48,617] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-10 21:51:48,647] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:51:48,675] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:51:48,746] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.22909ms
[2022-09-10 21:51:48,746] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7426b14f-5148-430a-9428-0a50449b9c66', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.22909ms', 'executionTime': '69.168572ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-10 21:51:48,747] - [tuq_helper:758] INFO - Create primary index
[2022-09-10 21:51:48,773] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-10 21:51:48,798] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-10 21:51:49,524] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 724.355145ms
[2022-09-10 21:51:49,524] - [tuq_helper:760] INFO - Check if index is online
[2022-09-10 21:51:49,584] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:51:49,629] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:51:49,637] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.893896ms
[2022-09-10 21:51:49,842] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:51:49,882] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-10 21:51:49,900] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-10 21:51:49,901] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-10 21:51:49,922] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-10 21:51:49,986] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:51:50,128] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:51:50,131] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:51:50,132] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:50,915] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:51:50,970] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:51:50,970] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-10 21:51:51,099] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:51:51,156] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:51:51,159] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:51:51,160] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:51,926] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:51:51,980] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-10 21:51:51,981] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-10 21:51:52,109] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-10 21:51:52,168] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-10 21:51:52,168] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-10 21:51:52,169] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:51:52,194] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-10 21:51:52,220] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-10 21:51:52,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.305061ms
[2022-09-10 21:51:52,257] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-10 21:51:52,282] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-10 21:51:52,340] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.215903ms
[2022-09-10 21:51:52,412] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-10 21:51:52,412] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.259649383191172, 'mem_free': 12777140224, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-10 21:51:52,413] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-10 21:51:52,417] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:51:52,417] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:53,209] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:51:53,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:51:53,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:54,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:51:54,488] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:51:54,488] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:55,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:51:55,835] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-10 21:51:55,835] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-10 21:51:57,207] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-10 21:52:04,976] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-10 21:52:05,127] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-10 21:52:06,374] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-10 21:52:06,402] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-10 21:52:06,402] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-10 21:52:06,462] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:52:06,515] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:52:06,567] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-10 21:52:06,568] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-10 21:52:06,647] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-10 21:52:06,648] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-10 21:52:06,674] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:52:06,802] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-10 21:52:06,803] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:52:06,829] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-10 21:52:06,855] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-10 21:52:06,855] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:52:06,881] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-10 21:52:06,907] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-10 21:52:06,907] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:52:06,933] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-10 21:52:06,959] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-10 21:52:06,959] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-10 21:52:06,985] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-10 21:52:06,986] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-10 21:52:06,986] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-10 21:52:06,987] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.126s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 24358 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-10.09.2022-20.03.cfg
Build Log: make-10.09.2022-20.03.log
Server Log: logs-10.09.2022-20.03.tar.gz

Finished