Building

Started building at 2022/09/11 02:07:44
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 11.09.2022-02.07
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at 0760e59, changes since last good build: 
fatal: Invalid revision range 67d5c96bc6a9c01fadbcc5dbdd2a62a39593b413..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable 1300ed66] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 4ce5e820] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable f34600c1] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable 607ea56d] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable 74bb868e] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 3b0fdccb] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable db3f2929] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable ac568643] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable b3837346] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 59d5b8e0] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable 96415f4e] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable d29c7794] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 77ba9f60] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable d0eae72] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable a2e06fd] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 1108506] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 9a62399] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 8a4fd23] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable 054cb78] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable 028ea19] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 01d3786] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/11 03:02:35
Testing mode: sanity,unit,functional,integration
Using storage type: fdb
Setting ulimit to 200000

Simple Test

Sep 11 03:07:39 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 03:11:32 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 03:12:17 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 11 03:13:17 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 11 03:14:07 b" 'stop_on_failure': 'True'}"
Sep 11 03:14:07 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 03:14:07 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_1'}"
Sep 11 03:14:07 b'-->result: '
Sep 11 03:14:07 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 11 03:14:07 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 03:14:07 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_2'}"
Sep 11 03:14:07 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 03:14:07 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 11 03:14:07 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_3'}"
Sep 11 03:14:07 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 03:14:07 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 03:14:07 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 11 03:14:07 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_4'}"
Sep 11 03:14:07 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 03:14:07 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 03:14:07 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 03:24:10 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 11 03:24:56 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 11 03:28:41 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 03:32:38 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 03:35:20 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 03:35:20 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_5'}"
Sep 11 03:35:20 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 03:35:20 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 11 03:35:20 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 11 03:35:20 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_6'}"
Sep 11 03:35:20 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 03:35:20 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 03:35:20 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 11 03:35:20 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_7'}"
Sep 11 03:35:20 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 03:35:20 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 03:35:20 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 11 03:35:20 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 11 03:35:20 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-02-57/test_8'}"
Sep 11 03:35:20 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 03:35:20 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 03:35:20 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 03:35:20 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 11 03:35:20 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 11 03:35:21 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 03:35:21 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 03:35:21 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 11 03:35:21 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 11 03:35:21 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 11 03:35:21 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 11 03:35:21 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 11 03:35:21 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=forestdb
Printing gsi_type=forestdb
gsi_type=forestdb
In here
-p makefile=True,gsi_type=forestdb
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'forestdb',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-11 03:38:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 03:38:15 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 03:38:21 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 03:38:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:38:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 03:38:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:38:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 03:38:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:38:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9003/pools/default error [Errno 111] Connection refused 
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 03:38:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:43 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-11 03:38:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '28', 'memoryTotal': 15466930176, 'memoryFree': 13631979520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:38:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 03:38:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:38:45 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:38:45 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1932316672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:38:45 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 03:38:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:38:46 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:38:46 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 13485465600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:38:46 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:38:47 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:38:47 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1705353216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:38:47 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:38:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:38:47 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:38:48 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 03:38:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 03:38:48 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-11 03:38:48 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:38:53 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 03:38:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 03:38:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-11 03:38:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 03:38:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 03:38:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 03:38:55 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 03:38:55 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-11 03:38:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:38:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 60.54660701044101, 'mem_free': 13416419328, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 46.74125123023987
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 03:38:59 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 03:39:04 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 03:39:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:05 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 03:39:05 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 03:39:07 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 03:39:07 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:39:07 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 03:39:07 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 03:39:37 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 03:39:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 03:39:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 03:39:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 120.223953ms
2022-09-11 03:39:37 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'fda767d1-4741-4dbb-bb41-a3b28400368d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '120.223953ms', 'executionTime': '120.043655ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 03:39:37 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 03:39:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 03:39:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 724.257322ms
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.943793ms
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 03:39:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.735015ms
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 03:39:39 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 58.519638ms
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.490366666003321, 'mem_free': 13367029760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:39:44 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 03:39:45 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 03:39:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 93.039s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.122975611271805, 'mem_free': 13363146752, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 03:39:50 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 03:39:50 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-11 03:39:51 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13338267648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-11 03:39:52 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:39:52 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13342846976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-11 03:39:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:39:53 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13343019008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:39:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:39:53 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13343748096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-11 03:39:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-11 03:39:54 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-11 03:39:54 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-11 03:39:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-11 03:40:00 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-11 03:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-11 03:40:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-11 03:40:01 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-11 03:40:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 03:40:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 03:40:03 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.41466814300006, 'mem_free': 13325078528, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.97083854675293
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-11 03:40:07 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-11 03:40:12 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-11 03:40:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:13 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-11 03:40:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-11 03:40:15 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-11 03:40:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:40:15 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-11 03:40:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-11 03:40:45 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-11 03:40:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 03:40:45 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 03:40:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 66.213657ms
2022-09-11 03:40:45 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'c8dc4572-abda-4245-8eb8-8eff16cc7a11', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.213657ms', 'executionTime': '66.150353ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 03:40:45 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-11 03:40:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-11 03:40:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-11 03:40:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 712.544108ms
2022-09-11 03:40:46 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-11 03:40:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 03:40:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 03:40:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.781777ms
2022-09-11 03:40:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:40:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-11 03:40:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-11 03:40:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-11 03:40:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-11 03:40:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:40:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeea0c8728fed9647389be88824ec55a253job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-11 03:40:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeea0c8728fed9647389be88824ec55a253job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-11 03:40:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 48.094077ms
2022-09-11 03:40:48 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeea0c8728fed9647389be88824ec55a253job_title) USING GSI
2022-09-11 03:40:49 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeea0c8728fed9647389be88824ec55a253job_title) USING GSI
2022-09-11 03:40:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeea0c8728fed9647389be88824ec55a253job_title%29+USING+GSI
2022-09-11 03:40:49 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 25.550086ms
2022-09-11 03:40:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea0c8728fed9647389be88824ec55a253job_title'
2022-09-11 03:40:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea0c8728fed9647389be88824ec55a253job_title%27
2022-09-11 03:40:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.879052ms
2022-09-11 03:40:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea0c8728fed9647389be88824ec55a253job_title'
2022-09-11 03:40:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea0c8728fed9647389be88824ec55a253job_title%27
2022-09-11 03:40:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.764815ms
2022-09-11 03:40:52 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 03:40:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 03:40:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-11 03:40:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.570131ms
2022-09-11 03:40:52 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '0db6003e-e7f4-43e5-b969-1eb8611b1a7a', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea0c8728fed9647389be88824ec55a253job_title', 'index_id': '99a1f8e71facaac5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.570131ms', 'executionTime': '2.449046ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-11 03:40:52 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 03:40:52 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 03:40:52 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 03:40:52 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-11 03:40:52 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-11 03:40:52 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-11 03:40:52 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-11 03:40:52 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-11 03:40:52 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-11 03:40:53 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 03:40:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-11 03:40:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-11 03:40:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 117.987894ms
2022-09-11 03:40:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-11 03:40:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-11 03:40:54 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 03:40:54 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea0c8728fed9647389be88824ec55a253job_title'
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea0c8728fed9647389be88824ec55a253job_title%27
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.337235ms
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeea0c8728fed9647389be88824ec55a253job_title ON default USING GSI
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeea0c8728fed9647389be88824ec55a253job_title+ON+default+USING+GSI
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 39.92708ms
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea0c8728fed9647389be88824ec55a253job_title'
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea0c8728fed9647389be88824ec55a253job_title%27
2022-09-11 03:40:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.797061ms
2022-09-11 03:40:55 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'a8bb8636-d8fa-4ddc-a8be-0f0ca3983294', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.797061ms', 'executionTime': '5.67372ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-11 03:40:55 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:40:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.026132ms
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 48.166317ms
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.33515214191106, 'mem_free': 13224206336, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-11 03:40:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-11 03:40:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-11 03:41:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 03:41:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-11 03:41:02 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-11 03:41:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.460s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_2'}
[2022-09-11 03:41:02,893] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:02,893] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:03,152] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:03,182] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:41:03,260] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:41:03,260] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-11 03:41:03,261] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:41:03,921] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:41:03,950] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13221216256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:41:03,979] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:41:03,979] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:41:03,980] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:41:04,018] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:41:04,052] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:41:04,052] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:41:04,079] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:41:04,080] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:41:04,081] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:41:04,081] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:41:04,129] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:41:04,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:04,132] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:04,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:04,379] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:41:04,449] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:41:04,450] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:41:04,479] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:41:04,480] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:41:04,506] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:41:04,588] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:41:04,615] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13221302272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:41:04,641] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:41:04,668] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:41:04,668] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:41:04,718] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:41:04,722] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:04,722] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:04,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:04,973] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:41:05,039] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:41:05,040] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:41:05,070] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:41:05,070] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:41:05,096] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:41:05,178] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:41:05,205] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13247098880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:41:05,231] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:41:05,259] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:41:05,259] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:41:05,309] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:41:05,312] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:05,313] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:05,566] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:05,567] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:41:05,632] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:41:05,633] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:41:05,662] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:41:05,663] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:41:05,688] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:41:05,773] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:41:05,800] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13219287040, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:41:05,826] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:41:05,853] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:41:05,854] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:41:05,905] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:41:05,907] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:05,907] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:06,151] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:06,152] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:41:06,220] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:41:06,221] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:41:06,252] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:41:06,253] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:41:06,278] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:41:06,333] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:41:06,953] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:41:11,956] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:41:12,038] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:41:12,042] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:12,042] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:12,307] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:12,308] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:41:12,379] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:41:12,380] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:41:12,380] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:41:13,447] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:41:13,501] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 03:41:13,501] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:41:14,397] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:41:14,729] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:41:15,009] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:41:15,074] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:41:15,147] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:15,147] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:15,416] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:15,420] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:15,421] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:15,662] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:15,667] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:15,667] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:15,912] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:15,917] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:15,917] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:16,306] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:19,816] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:41:19,817] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.70345052506165, 'mem_free': 13243920384, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:41:19,817] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:41:19,817] - [basetestcase:467] INFO - Time to execute basesetup : 16.926703453063965
[2022-09-11 03:41:19,869] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:41:19,869] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:41:19,922] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:41:19,922] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:41:19,975] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:41:19,975] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:41:20,026] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:41:20,027] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:41:20,079] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:41:20,142] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:41:20,142] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:41:20,143] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:41:25,151] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:41:25,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:41:25,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:41:25,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:41:26,353] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:41:26,519] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:41:28,183] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:41:28,202] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 03:41:28,841] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:41:28,841] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:41:28,842] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:41:58,871] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:41:58,900] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:41:58,925] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:41:58,992] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.557333ms
[2022-09-11 03:41:58,993] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b20f5db6-820a-4865-96cb-0c983760b939', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.557333ms', 'executionTime': '65.501095ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:41:58,993] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:41:59,018] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:41:59,043] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:41:59,722] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 677.679965ms
[2022-09-11 03:41:59,723] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:41:59,769] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:41:59,809] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:41:59,818] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.928206ms
[2022-09-11 03:42:00,045] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:42:00,084] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:42:00,097] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:42:00,099] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:42:00,113] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:42:00,196] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:42:01,004] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee23cb6c71d3bd41cb8617dd498f70878ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:42:01,031] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee23cb6c71d3bd41cb8617dd498f70878ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:42:01,080] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.736813ms
[2022-09-11 03:42:01,080] - [base_gsi:282] INFO - BUILD INDEX on default(employee23cb6c71d3bd41cb8617dd498f70878ajob_title) USING GSI
[2022-09-11 03:42:02,110] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee23cb6c71d3bd41cb8617dd498f70878ajob_title) USING GSI
[2022-09-11 03:42:02,136] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee23cb6c71d3bd41cb8617dd498f70878ajob_title%29+USING+GSI
[2022-09-11 03:42:02,163] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.22081ms
[2022-09-11 03:42:03,194] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee23cb6c71d3bd41cb8617dd498f70878ajob_title'
[2022-09-11 03:42:03,219] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee23cb6c71d3bd41cb8617dd498f70878ajob_title%27
[2022-09-11 03:42:03,227] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.875702ms
[2022-09-11 03:42:04,257] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee23cb6c71d3bd41cb8617dd498f70878ajob_title'
[2022-09-11 03:42:04,283] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee23cb6c71d3bd41cb8617dd498f70878ajob_title%27
[2022-09-11 03:42:04,290] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.619377ms
[2022-09-11 03:42:04,747] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 03:42:04,916] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:42:05,866] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:42:06,292] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:42:06,326] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:42:06,352] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 03:42:06,356] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.292732ms
[2022-09-11 03:42:06,356] - [task:3246] INFO - {'requestID': '4d01b3ec-50d4-487c-bf68-71fed33aa82b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee23cb6c71d3bd41cb8617dd498f70878ajob_title', 'index_id': '282e06a5191157ee', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.292732ms', 'executionTime': '2.23474ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 03:42:06,356] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:42:06,357] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:42:06,357] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:42:06,357] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:42:06,358] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 03:42:06,358] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:42:06,358] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:42:06,358] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:42:06,359] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:42:07,358] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:42:07,389] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:42:07,415] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 03:42:07,594] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 169.661401ms
[2022-09-11 03:42:07,594] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:42:07,594] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:42:08,369] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:42:08,369] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:42:09,401] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee23cb6c71d3bd41cb8617dd498f70878ajob_title'
[2022-09-11 03:42:09,426] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee23cb6c71d3bd41cb8617dd498f70878ajob_title%27
[2022-09-11 03:42:09,437] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.041489ms
[2022-09-11 03:42:09,463] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee23cb6c71d3bd41cb8617dd498f70878ajob_title ON default USING GSI
[2022-09-11 03:42:09,488] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee23cb6c71d3bd41cb8617dd498f70878ajob_title+ON+default+USING+GSI
[2022-09-11 03:42:09,535] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.280438ms
[2022-09-11 03:42:09,570] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee23cb6c71d3bd41cb8617dd498f70878ajob_title'
[2022-09-11 03:42:09,597] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee23cb6c71d3bd41cb8617dd498f70878ajob_title%27
[2022-09-11 03:42:09,605] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.737666ms
[2022-09-11 03:42:09,605] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd4c137ca-0425-4fe9-b3fa-2110404154af', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.737666ms', 'executionTime': '5.672847ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:42:09,715] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:42:09,718] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:09,719] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:10,054] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:10,111] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:42:10,111] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:42:10,172] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:42:10,228] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:42:10,231] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:10,231] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:10,577] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:10,636] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:42:10,636] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:42:10,699] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:42:10,756] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:42:10,757] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:42:10,757] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:42:10,784] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:42:10,811] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:42:10,821] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.201615ms
[2022-09-11 03:42:10,847] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:42:10,873] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:42:10,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.470139ms
[2022-09-11 03:42:10,982] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:42:10,982] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.8301426646385, 'mem_free': 13110628352, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:42:10,982] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:42:10,987] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:10,987] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:11,320] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:11,326] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:11,326] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:11,659] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:11,664] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:11,664] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:12,063] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:12,074] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:12,074] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:12,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:16,934] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:42:17,185] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:42:17,952] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:42:17,980] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:42:17,981] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:42:18,036] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:42:18,091] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:42:18,145] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:42:18,147] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:42:18,226] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:42:18,227] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:42:18,253] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:42:18,396] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:42:18,396] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:42:18,422] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:42:18,449] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:42:18,449] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:42:18,476] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:42:18,501] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:42:18,501] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:42:18,529] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:42:18,554] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:42:18,555] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:42:18,582] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:42:18,582] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:42:18,582] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:42:18,582] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 75.743s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_3'}
[2022-09-11 03:42:18,664] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:18,664] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:18,967] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:18,998] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:42:19,077] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:42:19,077] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-11 03:42:19,078] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:42:19,693] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:42:19,722] - [task:165] INFO -  {'uptime': '243', 'memoryTotal': 15466930176, 'memoryFree': 13182898176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:42:19,752] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:42:19,753] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:42:19,753] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:42:19,782] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:42:19,816] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:42:19,817] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:42:19,845] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:42:19,846] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:42:19,846] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:42:19,846] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:42:19,896] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:42:19,899] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:19,900] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:20,203] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:20,204] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:42:20,278] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:42:20,279] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:42:20,309] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:42:20,309] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:42:20,335] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:42:20,418] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:42:20,445] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13184008192, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:42:20,470] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:42:20,499] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:42:20,499] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:42:20,549] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:42:20,553] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:20,554] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:20,849] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:20,850] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:42:20,924] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:42:20,925] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:42:20,954] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:42:20,955] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:42:20,981] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:42:21,064] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:42:21,090] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13183848448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:42:21,115] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:42:21,142] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:42:21,143] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:42:21,192] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:42:21,195] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:21,195] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:21,498] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:21,500] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:42:21,574] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:42:21,575] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:42:21,604] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:42:21,605] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:42:21,630] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:42:21,713] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:42:21,741] - [task:165] INFO -  {'uptime': '236', 'memoryTotal': 15466930176, 'memoryFree': 13184258048, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:42:21,767] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:42:21,797] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:42:21,798] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:42:21,851] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:42:21,854] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:21,855] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:22,157] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:22,159] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:42:22,232] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:42:22,234] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:42:22,264] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:42:22,265] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:42:22,290] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:42:22,349] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:42:22,900] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:42:27,905] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:42:27,988] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:42:27,994] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:27,994] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:28,313] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:28,314] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:42:28,388] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:42:28,389] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:42:28,389] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:42:29,464] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:42:29,522] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 03:42:29,523] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:42:30,427] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:42:30,742] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:42:31,027] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:42:31,029] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:42:31,092] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:31,092] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:31,407] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:31,412] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:31,412] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:31,707] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:31,712] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:31,712] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:32,014] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:32,020] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:32,020] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:32,565] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:36,543] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:42:36,544] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.31714327914948, 'mem_free': 13191221248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:42:36,544] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:42:36,544] - [basetestcase:467] INFO - Time to execute basesetup : 17.882208824157715
[2022-09-11 03:42:36,595] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:42:36,596] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:42:36,649] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:42:36,649] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:42:36,703] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:42:36,703] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:42:36,757] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:42:36,757] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:42:36,808] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:42:36,866] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:42:36,866] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:42:36,866] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:42:41,874] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:42:41,878] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:42:41,878] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:42:42,207] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:42:43,237] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:42:43,408] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:42:45,963] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:42:45,980] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 03:42:46,818] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:42:46,818] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:42:46,818] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:43:16,830] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:43:16,859] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:43:16,884] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:43:16,953] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.927164ms
[2022-09-11 03:43:16,954] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b811fd07-c868-41bc-8c4e-05baef8f8327', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.927164ms', 'executionTime': '66.851895ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:43:16,954] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:43:16,982] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:43:17,008] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:43:17,698] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 687.790512ms
[2022-09-11 03:43:17,698] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:43:17,770] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:43:17,804] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:43:17,812] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.279324ms
[2022-09-11 03:43:18,038] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:43:18,077] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:43:18,092] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:43:18,092] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:43:18,114] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:43:18,182] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:43:18,995] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeefdfb8fe11edb4993b1d20cc8aa73b417job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:43:19,021] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeefdfb8fe11edb4993b1d20cc8aa73b417job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:43:19,066] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.142727ms
[2022-09-11 03:43:19,067] - [base_gsi:282] INFO - BUILD INDEX on default(employeefdfb8fe11edb4993b1d20cc8aa73b417job_title) USING GSI
[2022-09-11 03:43:20,098] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeefdfb8fe11edb4993b1d20cc8aa73b417job_title) USING GSI
[2022-09-11 03:43:20,124] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeefdfb8fe11edb4993b1d20cc8aa73b417job_title%29+USING+GSI
[2022-09-11 03:43:20,149] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.025858ms
[2022-09-11 03:43:21,180] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdfb8fe11edb4993b1d20cc8aa73b417job_title'
[2022-09-11 03:43:21,206] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdfb8fe11edb4993b1d20cc8aa73b417job_title%27
[2022-09-11 03:43:21,216] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.277861ms
[2022-09-11 03:43:22,245] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdfb8fe11edb4993b1d20cc8aa73b417job_title'
[2022-09-11 03:43:22,270] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdfb8fe11edb4993b1d20cc8aa73b417job_title%27
[2022-09-11 03:43:22,279] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.097412ms
[2022-09-11 03:43:22,752] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 03:43:22,923] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:43:23,604] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:43:24,281] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:43:24,313] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:43:24,339] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 03:43:24,344] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.957758ms
[2022-09-11 03:43:24,344] - [task:3246] INFO - {'requestID': '25cdc802-c5c5-4e51-b4f9-c6af0782dba3', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeefdfb8fe11edb4993b1d20cc8aa73b417job_title', 'index_id': '4d69dc7de864e8b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.957758ms', 'executionTime': '2.897589ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-09-11 03:43:24,344] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:43:24,345] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:43:24,345] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:43:24,345] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:43:24,346] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 03:43:24,346] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:43:24,346] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:43:24,346] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:43:24,347] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:43:25,346] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:43:25,376] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:43:25,402] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 03:43:25,535] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 123.316452ms
[2022-09-11 03:43:25,535] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:43:25,536] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:43:26,306] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:43:26,307] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:43:27,337] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdfb8fe11edb4993b1d20cc8aa73b417job_title'
[2022-09-11 03:43:27,362] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdfb8fe11edb4993b1d20cc8aa73b417job_title%27
[2022-09-11 03:43:27,372] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.418967ms
[2022-09-11 03:43:27,398] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeefdfb8fe11edb4993b1d20cc8aa73b417job_title ON default USING GSI
[2022-09-11 03:43:27,423] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeefdfb8fe11edb4993b1d20cc8aa73b417job_title+ON+default+USING+GSI
[2022-09-11 03:43:27,462] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.597388ms
[2022-09-11 03:43:27,496] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefdfb8fe11edb4993b1d20cc8aa73b417job_title'
[2022-09-11 03:43:27,525] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefdfb8fe11edb4993b1d20cc8aa73b417job_title%27
[2022-09-11 03:43:27,534] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.025177ms
[2022-09-11 03:43:27,534] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8902b839-3553-487e-b054-4e463b08bb23', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.025177ms', 'executionTime': '6.936708ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:43:27,643] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:43:27,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:27,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:28,046] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:28,103] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:43:28,103] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:43:28,170] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:43:28,225] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:43:28,228] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:28,228] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:28,606] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:28,664] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:43:28,664] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:43:28,733] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:43:28,790] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:43:28,790] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:43:28,790] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:43:28,816] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:43:28,843] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:43:28,853] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.350061ms
[2022-09-11 03:43:28,881] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:43:28,908] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:43:28,962] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.805247ms
[2022-09-11 03:43:29,025] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:43:29,025] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.201778034418199, 'mem_free': 13078695936, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:43:29,025] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:43:29,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:29,032] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:29,411] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:29,416] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:29,417] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:29,952] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:29,958] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:29,959] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:30,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:30,638] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:30,638] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:31,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:35,592] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:43:35,733] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:43:37,028] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:43:37,055] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:43:37,055] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:43:37,114] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:43:37,174] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:43:37,229] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:43:37,230] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:43:37,312] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:43:37,313] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:43:37,340] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:43:37,472] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:43:37,472] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:43:37,498] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:43:37,524] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:43:37,524] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:43:37,552] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:43:37,577] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:43:37,577] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:43:37,604] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:43:37,629] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:43:37,630] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:43:37,657] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:43:37,657] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:43:37,657] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:43:37,658] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.047s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_4'}
[2022-09-11 03:43:37,759] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:37,759] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:38,140] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:38,170] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:43:38,250] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:43:38,251] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-11 03:43:38,251] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:43:38,788] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:43:38,815] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13071769600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:43:38,841] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:43:38,841] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:43:38,842] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:43:38,878] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:43:38,918] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:43:38,919] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:43:38,946] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:43:38,947] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:43:38,947] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:43:38,947] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:43:38,994] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:43:38,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:38,998] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:39,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:39,360] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:43:39,441] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:43:39,442] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:43:39,472] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:43:39,473] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:43:39,498] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:43:39,588] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:43:39,614] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13143117824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:43:39,640] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:43:39,668] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:43:39,668] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:43:39,718] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:43:39,721] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:39,721] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:40,097] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:40,098] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:43:40,180] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:43:40,181] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:43:40,210] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:43:40,211] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:43:40,237] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:43:40,319] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:43:40,345] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13142888448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:43:40,371] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:43:40,400] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:43:40,401] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:43:40,451] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:43:40,456] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:40,456] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:40,821] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:40,822] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:43:40,905] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:43:40,907] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:43:40,936] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:43:40,937] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:43:40,962] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:43:41,046] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:43:41,072] - [task:165] INFO -  {'uptime': '316', 'memoryTotal': 15466930176, 'memoryFree': 13143068672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:43:41,097] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:43:41,124] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:43:41,124] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:43:41,175] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:43:41,178] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:41,178] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:41,545] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:41,546] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:43:41,627] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:43:41,628] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:43:41,657] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:43:41,658] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:43:41,683] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:43:41,739] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:43:42,261] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:43:47,265] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:43:47,349] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:43:47,352] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:47,353] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:47,734] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:47,735] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:43:47,819] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:43:47,821] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:43:47,821] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:43:48,888] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:43:48,946] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 03:43:48,947] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:43:49,842] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:43:50,168] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:43:50,465] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:43:50,468] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:43:50,529] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:50,530] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:50,901] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:50,906] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:50,906] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:51,258] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:51,263] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:51,263] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:51,732] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:51,741] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:43:51,742] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:43:52,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:43:56,910] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:43:56,911] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.91640523720249, 'mem_free': 13146607616, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:43:56,911] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:43:56,911] - [basetestcase:467] INFO - Time to execute basesetup : 19.154690980911255
[2022-09-11 03:43:56,961] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:43:56,961] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:43:57,013] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:43:57,014] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:43:57,065] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:43:57,066] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:43:57,118] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:43:57,118] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:43:57,174] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:43:57,238] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:43:57,238] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:43:57,239] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:44:02,247] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:44:02,251] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:02,251] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:02,649] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:03,702] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:44:03,870] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:44:06,305] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:44:06,320] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-11 03:44:06,992] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:44:06,992] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:44:06,992] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:44:37,022] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:44:37,051] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:44:37,077] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:44:37,150] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.709104ms
[2022-09-11 03:44:37,150] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4f78927c-7ca4-4ec5-86fd-360b4c546db7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.709104ms', 'executionTime': '70.649481ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:44:37,151] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:44:37,179] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:44:37,206] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:44:37,921] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 713.203963ms
[2022-09-11 03:44:37,921] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:44:37,984] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:44:38,018] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:44:38,026] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.648426ms
[2022-09-11 03:44:38,231] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:44:38,267] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:44:38,281] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:44:38,281] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:44:38,303] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:44:38,373] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:44:39,193] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee2298abede4b240749033b0a56bfb0810job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:44:39,218] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee2298abede4b240749033b0a56bfb0810job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:44:39,278] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.028673ms
[2022-09-11 03:44:39,278] - [base_gsi:282] INFO - BUILD INDEX on default(employee2298abede4b240749033b0a56bfb0810job_title) USING GSI
[2022-09-11 03:44:40,312] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee2298abede4b240749033b0a56bfb0810job_title) USING GSI
[2022-09-11 03:44:40,339] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee2298abede4b240749033b0a56bfb0810job_title%29+USING+GSI
[2022-09-11 03:44:40,366] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.183666ms
[2022-09-11 03:44:41,401] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2298abede4b240749033b0a56bfb0810job_title'
[2022-09-11 03:44:41,428] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2298abede4b240749033b0a56bfb0810job_title%27
[2022-09-11 03:44:41,438] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.316807ms
[2022-09-11 03:44:42,470] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2298abede4b240749033b0a56bfb0810job_title'
[2022-09-11 03:44:42,496] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2298abede4b240749033b0a56bfb0810job_title%27
[2022-09-11 03:44:42,505] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.008818ms
[2022-09-11 03:44:42,991] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 03:44:43,165] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:44:43,989] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:44:44,177] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-11 03:44:44,177] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:44:44,354] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-11 03:44:44,354] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-11 03:44:44,355] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2471161928, 0, b'')
[2022-09-11 03:44:44,507] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:44:44,535] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:44:44,561] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 03:44:44,566] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.076218ms
[2022-09-11 03:44:44,566] - [task:3246] INFO - {'requestID': '06eb290f-1693-49ff-9749-f7d40686505e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2298abede4b240749033b0a56bfb0810job_title', 'index_id': '106fa73f035b8bb', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.076218ms', 'executionTime': '2.999921ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-09-11 03:44:44,566] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:44:44,566] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:44:44,567] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:44:44,567] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:44:44,567] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 03:44:44,568] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:44:44,568] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:44:44,568] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:44:44,568] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:44:45,568] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:44:45,598] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:44:45,624] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 03:44:45,756] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 124.138547ms
[2022-09-11 03:44:45,756] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:44:45,756] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:44:46,541] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:44:46,542] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:44:47,574] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2298abede4b240749033b0a56bfb0810job_title'
[2022-09-11 03:44:47,600] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2298abede4b240749033b0a56bfb0810job_title%27
[2022-09-11 03:44:47,610] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.692959ms
[2022-09-11 03:44:47,635] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee2298abede4b240749033b0a56bfb0810job_title ON default USING GSI
[2022-09-11 03:44:47,661] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee2298abede4b240749033b0a56bfb0810job_title+ON+default+USING+GSI
[2022-09-11 03:44:47,703] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.373735ms
[2022-09-11 03:44:47,740] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2298abede4b240749033b0a56bfb0810job_title'
[2022-09-11 03:44:47,766] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2298abede4b240749033b0a56bfb0810job_title%27
[2022-09-11 03:44:47,776] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.821953ms
[2022-09-11 03:44:47,776] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6b709a18-4817-4fec-9c93-e1d8ea659437', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.821953ms', 'executionTime': '5.745052ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:44:47,880] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:44:47,883] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:47,884] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:48,364] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:48,422] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:44:48,423] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:44:48,500] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:44:48,558] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:44:48,561] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:48,561] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:49,017] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:49,075] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:44:49,076] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:44:49,152] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:44:49,208] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:44:49,209] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:44:49,209] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:44:49,235] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:44:49,260] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:44:49,271] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.192713ms
[2022-09-11 03:44:49,297] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:44:49,323] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:44:49,374] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.310799ms
[2022-09-11 03:44:49,439] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:44:49,439] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.321980456323134, 'mem_free': 13026746368, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:44:49,439] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:44:49,444] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:49,444] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:49,881] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:49,886] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:49,886] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:50,563] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:50,571] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:50,571] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:51,364] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:51,376] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:51,376] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:52,151] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:56,930] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:44:57,075] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:44:57,445] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:44:57,472] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:44:57,473] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:44:57,531] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:44:57,586] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:44:57,638] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:44:57,639] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:44:57,719] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:44:57,720] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:44:57,745] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:44:57,874] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:44:57,874] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:44:57,901] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:44:57,926] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:44:57,926] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:44:57,952] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:44:57,977] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:44:57,978] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:44:58,003] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:44:58,029] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:44:58,029] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:44:58,055] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:44:58,056] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:44:58,056] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:44:58,056] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.352s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_5'}
[2022-09-11 03:44:58,159] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:58,159] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:58,603] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:58,634] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:44:58,718] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:44:58,718] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-11 03:44:58,719] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:44:59,184] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:44:59,211] - [task:165] INFO -  {'uptime': '403', 'memoryTotal': 15466930176, 'memoryFree': 13102260224, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:44:59,237] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:44:59,238] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:44:59,238] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:44:59,273] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:44:59,314] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:44:59,315] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:44:59,344] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:44:59,345] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:44:59,345] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:44:59,345] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:44:59,393] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:44:59,396] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:44:59,396] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:44:59,828] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:44:59,829] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:44:59,926] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:44:59,927] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:44:59,958] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:44:59,959] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:44:59,987] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:45:00,077] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:45:00,105] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13099057152, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:45:00,130] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:45:00,158] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:45:00,159] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:45:00,209] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:45:00,212] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:00,212] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:00,655] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:00,656] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:45:00,749] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:45:00,750] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:45:00,781] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:45:00,781] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:45:00,807] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:45:00,892] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:45:00,917] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13098803200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:45:00,944] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:45:00,972] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:45:00,972] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:45:01,023] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:45:01,029] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:01,029] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:01,493] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:01,494] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:45:01,594] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:45:01,595] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:45:01,627] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:45:01,628] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:45:01,655] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:45:01,746] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:45:01,776] - [task:165] INFO -  {'uptime': '397', 'memoryTotal': 15466930176, 'memoryFree': 13100335104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:45:01,804] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:45:01,834] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:45:01,835] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:45:01,888] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:45:01,891] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:01,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:02,342] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:02,344] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:45:02,436] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:45:02,437] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:45:02,469] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:45:02,470] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:45:02,495] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:45:02,553] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:45:03,162] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:45:08,167] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:45:08,250] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:45:08,254] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:08,255] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:08,680] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:08,682] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:45:08,771] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:45:08,772] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:45:08,773] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:45:09,670] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:45:09,726] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 03:45:09,726] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:45:10,637] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:45:10,953] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:45:11,233] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:45:11,236] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:45:11,298] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:11,298] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:11,730] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:11,735] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:11,735] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:12,135] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:12,140] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:12,140] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:12,865] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:12,877] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:12,877] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:13,666] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:18,634] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:45:18,635] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.45811329240118, 'mem_free': 13106753536, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:45:18,635] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:45:18,635] - [basetestcase:467] INFO - Time to execute basesetup : 20.478811979293823
[2022-09-11 03:45:18,686] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:45:18,686] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:45:18,737] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:45:18,738] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:45:18,790] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:45:18,790] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:45:18,843] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:45:18,843] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:45:18,895] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:45:18,954] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:45:18,954] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:45:18,955] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:45:23,963] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:45:23,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:45:23,967] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:45:24,410] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:45:25,459] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:45:25,634] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:45:28,027] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:45:28,047] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 03:45:29,098] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:45:29,099] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:45:29,099] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:45:59,121] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:45:59,151] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:45:59,176] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:45:59,244] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.1232ms
[2022-09-11 03:45:59,245] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ceb4b2a1-5478-4aef-8604-407b35277542', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.1232ms', 'executionTime': '66.056843ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:45:59,245] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:45:59,272] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:45:59,298] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:45:59,990] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 686.144756ms
[2022-09-11 03:45:59,990] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:46:00,041] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:46:00,081] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:46:00,089] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.877925ms
[2022-09-11 03:46:00,290] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:46:00,326] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:46:00,342] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:46:00,342] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:46:00,357] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:46:00,424] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:46:01,250] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeec5d3e77d9b8475cad636794aa7d3551job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:46:01,276] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeec5d3e77d9b8475cad636794aa7d3551job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:46:01,327] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.890076ms
[2022-09-11 03:46:01,328] - [base_gsi:282] INFO - BUILD INDEX on default(employeeec5d3e77d9b8475cad636794aa7d3551job_title) USING GSI
[2022-09-11 03:46:02,358] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeec5d3e77d9b8475cad636794aa7d3551job_title) USING GSI
[2022-09-11 03:46:02,383] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeec5d3e77d9b8475cad636794aa7d3551job_title%29+USING+GSI
[2022-09-11 03:46:02,406] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.910414ms
[2022-09-11 03:46:03,437] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeec5d3e77d9b8475cad636794aa7d3551job_title'
[2022-09-11 03:46:03,463] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeec5d3e77d9b8475cad636794aa7d3551job_title%27
[2022-09-11 03:46:03,470] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.931842ms
[2022-09-11 03:46:04,502] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeec5d3e77d9b8475cad636794aa7d3551job_title'
[2022-09-11 03:46:04,527] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeec5d3e77d9b8475cad636794aa7d3551job_title%27
[2022-09-11 03:46:04,534] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.671459ms
[2022-09-11 03:46:05,014] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:46:05,181] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:46:08,047] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:46:08,539] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:46:08,571] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:46:08,596] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 03:46:08,601] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.916892ms
[2022-09-11 03:46:08,601] - [task:3246] INFO - {'requestID': 'd76868da-f975-4acd-bf8d-049a3fa6bcc6', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeec5d3e77d9b8475cad636794aa7d3551job_title', 'index_id': 'ec266fa59c3d26', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.916892ms', 'executionTime': '2.846257ms', 'resultCount': 1, 'resultSize': 722, 'serviceLoad': 6}}
[2022-09-11 03:46:08,601] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:46:08,601] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:46:08,603] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:46:08,603] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:46:08,603] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 03:46:08,603] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:46:08,604] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:46:08,604] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:46:08,604] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:46:09,604] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:46:09,635] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:46:09,662] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 03:46:09,842] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 165.897989ms
[2022-09-11 03:46:09,842] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:46:09,843] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:46:11,386] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:46:11,386] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:46:12,417] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeec5d3e77d9b8475cad636794aa7d3551job_title'
[2022-09-11 03:46:12,442] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeec5d3e77d9b8475cad636794aa7d3551job_title%27
[2022-09-11 03:46:12,452] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.640398ms
[2022-09-11 03:46:12,478] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeec5d3e77d9b8475cad636794aa7d3551job_title ON default USING GSI
[2022-09-11 03:46:12,503] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeec5d3e77d9b8475cad636794aa7d3551job_title+ON+default+USING+GSI
[2022-09-11 03:46:12,559] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.302024ms
[2022-09-11 03:46:12,594] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeec5d3e77d9b8475cad636794aa7d3551job_title'
[2022-09-11 03:46:12,624] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeec5d3e77d9b8475cad636794aa7d3551job_title%27
[2022-09-11 03:46:12,632] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.785239ms
[2022-09-11 03:46:12,632] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '71f49189-e941-4d23-aa8c-d4f63e5c6368', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.785239ms', 'executionTime': '5.70266ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:46:12,743] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:46:12,748] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:12,749] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:13,309] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:13,364] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:46:13,365] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:46:13,454] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:46:13,510] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:46:13,514] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:13,514] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:14,080] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:14,142] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:46:14,142] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:46:14,231] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:46:14,290] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:46:14,291] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:46:14,291] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:46:14,318] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:46:14,344] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:46:14,355] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.32959ms
[2022-09-11 03:46:14,382] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:46:14,408] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:46:14,464] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.953843ms
[2022-09-11 03:46:14,528] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:46:14,528] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.38552239961076, 'mem_free': 12963340288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:46:14,529] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:46:14,534] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:14,534] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:15,094] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:15,100] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:15,100] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:15,617] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:15,622] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:15,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:16,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:16,427] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:16,427] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:17,387] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:23,652] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:46:23,797] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:46:24,278] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:46:24,306] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:46:24,306] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:46:24,363] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:46:24,421] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:46:24,478] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:46:24,479] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:46:24,563] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:46:24,564] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:46:24,591] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:46:24,723] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:46:24,723] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:46:24,750] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:46:24,778] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:46:24,778] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:46:24,804] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:46:24,831] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:46:24,831] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:46:24,858] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:46:24,884] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:46:24,884] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:46:24,915] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:46:24,915] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:46:24,915] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:46:24,916] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 86.827s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_6'}
[2022-09-11 03:46:25,040] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:25,040] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:25,538] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:25,571] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:46:25,652] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:46:25,652] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-11 03:46:25,653] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:46:26,067] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:46:26,095] - [task:165] INFO -  {'uptime': '489', 'memoryTotal': 15466930176, 'memoryFree': 13042425856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:46:26,121] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:46:26,121] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:46:26,122] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:46:26,158] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:46:26,194] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:46:26,194] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:46:26,223] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:46:26,224] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:46:26,224] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:46:26,224] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:46:26,274] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:46:26,277] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:26,277] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:26,790] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:26,791] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:46:26,896] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:46:26,898] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:46:26,928] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:46:26,929] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:46:26,955] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:46:27,041] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:46:27,067] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13041225728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:46:27,094] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:46:27,122] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:46:27,123] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:46:27,177] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:46:27,181] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:27,181] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:27,696] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:27,697] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:46:27,804] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:46:27,805] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:46:27,835] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:46:27,836] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:46:27,862] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:46:27,952] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:46:27,979] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13040590848, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:46:28,005] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:46:28,034] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:46:28,034] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:46:28,086] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:46:28,092] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:28,092] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:28,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:28,592] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:46:28,700] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:46:28,701] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:46:28,731] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:46:28,732] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:46:28,757] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:46:28,843] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:46:28,871] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13040836608, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:46:28,897] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:46:28,929] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:46:28,930] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:46:28,980] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:46:28,983] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:28,983] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:29,502] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:29,503] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:46:29,608] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:46:29,609] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:46:29,640] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:46:29,641] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:46:29,666] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:46:29,723] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:46:30,389] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:46:35,393] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:46:35,476] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:46:35,481] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:35,481] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:36,003] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:36,004] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:46:36,103] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:46:36,104] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:46:36,105] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:46:36,883] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:46:36,938] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 03:46:36,939] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:46:37,956] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:46:38,262] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:46:38,524] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:46:38,527] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:46:38,579] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:38,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:39,101] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:39,106] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:39,106] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:39,738] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:39,746] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:39,746] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:40,614] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:40,622] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:40,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:41,512] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:46,853] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:46:46,853] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.66426935841908, 'mem_free': 13036343296, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:46:46,854] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:46:46,854] - [basetestcase:467] INFO - Time to execute basesetup : 21.816384077072144
[2022-09-11 03:46:46,904] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:46:46,905] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:46:46,958] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:46:46,959] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:46:47,011] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:46:47,012] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:46:47,063] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:46:47,064] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:46:47,115] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:46:47,173] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:46:47,173] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:46:47,175] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:46:52,184] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:46:52,188] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:46:52,188] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:46:52,676] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:46:53,784] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:46:53,952] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:46:56,358] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:46:56,376] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-11 03:46:57,865] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:46:57,865] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:46:57,865] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:47:27,893] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:47:27,922] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:47:27,947] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:47:28,014] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.258879ms
[2022-09-11 03:47:28,014] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'dcc792e9-75d6-4580-b1e3-701546238c13', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.258879ms', 'executionTime': '65.189191ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:47:28,014] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:47:28,040] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:47:28,064] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:47:28,719] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 652.365763ms
[2022-09-11 03:47:28,719] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:47:28,797] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:47:28,840] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:47:28,859] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.472904ms
[2022-09-11 03:47:29,068] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:47:29,104] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:47:29,117] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:47:29,117] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:47:29,128] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:47:29,197] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:47:30,028] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee54f10b6ebec3494f95631ddcd7ed7149job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:47:30,054] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee54f10b6ebec3494f95631ddcd7ed7149job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:47:30,134] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 78.785994ms
[2022-09-11 03:47:30,135] - [base_gsi:282] INFO - BUILD INDEX on default(employee54f10b6ebec3494f95631ddcd7ed7149job_title) USING GSI
[2022-09-11 03:47:31,165] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee54f10b6ebec3494f95631ddcd7ed7149job_title) USING GSI
[2022-09-11 03:47:31,190] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee54f10b6ebec3494f95631ddcd7ed7149job_title%29+USING+GSI
[2022-09-11 03:47:31,212] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.779372ms
[2022-09-11 03:47:32,242] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee54f10b6ebec3494f95631ddcd7ed7149job_title'
[2022-09-11 03:47:32,266] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee54f10b6ebec3494f95631ddcd7ed7149job_title%27
[2022-09-11 03:47:32,275] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.596407ms
[2022-09-11 03:47:33,306] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee54f10b6ebec3494f95631ddcd7ed7149job_title'
[2022-09-11 03:47:33,331] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee54f10b6ebec3494f95631ddcd7ed7149job_title%27
[2022-09-11 03:47:33,341] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.201623ms
[2022-09-11 03:47:33,724] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-11 03:47:33,893] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:47:35,033] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:47:35,650] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-11 03:47:35,824] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:47:37,129] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:47:37,763] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:47:37,945] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:47:41,447] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:47:42,378] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:47:42,409] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:47:42,434] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-11 03:47:42,438] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.231801ms
[2022-09-11 03:47:42,438] - [task:3246] INFO - {'requestID': '65bef85c-97cc-4353-a145-fa833574a1b5', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee54f10b6ebec3494f95631ddcd7ed7149job_title', 'index_id': 'c7fa31b07bcc4946', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.231801ms', 'executionTime': '2.17457ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-11 03:47:42,439] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:47:42,439] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:47:42,439] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:47:42,440] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:47:42,440] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-11 03:47:42,440] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:47:42,441] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:47:42,441] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:47:42,441] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-11 03:47:43,440] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:47:43,469] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-11 03:47:43,494] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-11 03:47:43,705] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 194.967993ms
[2022-09-11 03:47:43,705] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:47:43,706] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:47:45,246] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:47:45,246] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-11 03:47:46,277] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee54f10b6ebec3494f95631ddcd7ed7149job_title'
[2022-09-11 03:47:46,303] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee54f10b6ebec3494f95631ddcd7ed7149job_title%27
[2022-09-11 03:47:46,312] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.245913ms
[2022-09-11 03:47:46,338] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee54f10b6ebec3494f95631ddcd7ed7149job_title ON default USING GSI
[2022-09-11 03:47:46,362] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee54f10b6ebec3494f95631ddcd7ed7149job_title+ON+default+USING+GSI
[2022-09-11 03:47:46,397] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.882166ms
[2022-09-11 03:47:46,438] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee54f10b6ebec3494f95631ddcd7ed7149job_title'
[2022-09-11 03:47:46,463] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee54f10b6ebec3494f95631ddcd7ed7149job_title%27
[2022-09-11 03:47:46,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.798029ms
[2022-09-11 03:47:46,471] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '92c6e814-03f3-42d7-9693-15f3e82a844d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.798029ms', 'executionTime': '5.73917ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:47:46,578] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:47:46,581] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:47:46,582] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:47:47,227] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:47:47,282] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:47:47,282] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:47:47,387] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:47:47,447] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:47:47,450] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:47:47,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:47:48,088] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:47:48,144] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:47:48,145] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:47:48,252] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:47:48,311] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:47:48,311] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:47:48,312] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:47:48,337] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:47:48,362] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:47:48,369] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.793046ms
[2022-09-11 03:47:48,396] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:47:48,421] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:47:48,453] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.523473ms
[2022-09-11 03:47:48,538] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:47:48,538] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.18450795791481, 'mem_free': 12839424000, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:47:48,538] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:47:48,543] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:47:48,543] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:47:49,169] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:47:49,174] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:47:49,175] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:47:50,178] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:47:50,186] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:47:50,186] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:47:51,278] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:47:51,286] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:47:51,286] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:47:52,414] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:47:58,891] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:47:59,037] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:47:59,415] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:47:59,443] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:47:59,443] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:47:59,500] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:47:59,553] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:47:59,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:47:59,608] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:47:59,688] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:47:59,689] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:47:59,716] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:47:59,848] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:47:59,848] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:47:59,875] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:47:59,900] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:47:59,900] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:47:59,927] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:47:59,952] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:47:59,952] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:47:59,979] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:48:00,004] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:48:00,004] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:48:00,031] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:48:00,031] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-11 03:48:00,032] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:48:00,032] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 95.050s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_7'}
[2022-09-11 03:48:00,188] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:00,188] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:00,798] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:00,832] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:48:00,913] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:48:00,914] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-11 03:48:00,914] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:48:01,215] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:48:01,244] - [task:165] INFO -  {'uptime': '584', 'memoryTotal': 15466930176, 'memoryFree': 12974505984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:48:01,271] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:48:01,271] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:48:01,272] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:48:01,307] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:48:01,339] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:48:01,340] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:48:01,369] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:48:01,370] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:48:01,370] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:48:01,371] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:48:01,422] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:48:01,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:01,425] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:02,067] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:02,068] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:48:02,192] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:48:02,193] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:48:02,226] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:48:02,226] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:48:02,254] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:48:02,340] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:48:02,367] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 12971028480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:48:02,393] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:48:02,422] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:48:02,423] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:48:02,473] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:48:02,476] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:02,476] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:03,087] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:03,088] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:48:03,203] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:48:03,204] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:48:03,233] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:48:03,233] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:48:03,258] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:48:03,344] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:48:03,370] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 12971651072, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:48:03,396] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:48:03,423] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:48:03,423] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:48:03,473] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:48:03,479] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:03,479] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:04,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:04,088] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:48:04,205] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:48:04,206] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:48:04,235] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:48:04,236] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:48:04,261] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:48:04,345] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:48:04,371] - [task:165] INFO -  {'uptime': '582', 'memoryTotal': 15466930176, 'memoryFree': 12971855872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:48:04,397] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:48:04,429] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:48:04,430] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:48:04,482] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:48:04,485] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:04,485] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:05,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:05,079] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:48:05,193] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:48:05,194] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:48:05,224] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:48:05,225] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:48:05,251] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:48:05,308] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:48:05,819] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:48:10,822] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:48:10,906] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:48:10,911] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:10,912] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:11,508] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:11,509] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:48:11,619] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:48:11,620] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:48:11,620] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:48:12,437] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:48:12,492] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 03:48:12,493] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:48:13,396] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:48:13,696] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:48:13,955] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:48:13,959] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 03:48:14,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:14,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:14,579] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:14,584] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:14,584] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:15,377] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:15,385] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:15,385] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:16,435] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:16,449] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:16,449] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:17,473] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:23,870] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:48:23,871] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.55855843335326, 'mem_free': 12989472768, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:48:23,871] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:48:23,871] - [basetestcase:467] INFO - Time to execute basesetup : 23.686002492904663
[2022-09-11 03:48:23,922] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:48:23,922] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:48:23,979] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:48:23,979] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:48:24,031] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:48:24,032] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:48:24,086] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:48:24,087] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:48:24,139] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:48:24,198] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:48:24,198] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:48:24,198] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:48:29,210] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:48:29,214] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:48:29,214] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:48:29,787] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:48:30,817] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:48:31,089] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:48:33,307] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:48:33,395] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:48:33,395] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:48:33,395] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:49:03,406] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:49:03,435] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:49:03,460] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:49:03,528] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.433162ms
[2022-09-11 03:49:03,528] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4c05ab05-1366-4ff6-8ab6-4992ba5defe1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.433162ms', 'executionTime': '65.359881ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:49:03,528] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:49:03,555] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:49:03,584] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:49:04,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 643.751684ms
[2022-09-11 03:49:04,230] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:49:04,290] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:49:04,327] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:49:04,335] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.674325ms
[2022-09-11 03:49:04,535] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:49:04,573] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:49:04,592] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:49:04,595] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:49:04,611] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:49:04,612] - [base_gsi:326] INFO - []
[2022-09-11 03:49:05,496] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea8b0b7da499e49c39f9bb8566f466ea9job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:49:05,521] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeea8b0b7da499e49c39f9bb8566f466ea9job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:49:05,599] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 76.02827ms
[2022-09-11 03:49:05,634] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea8b0b7da499e49c39f9bb8566f466ea9join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:49:05,661] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeea8b0b7da499e49c39f9bb8566f466ea9join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:49:05,716] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.405654ms
[2022-09-11 03:49:05,716] - [base_gsi:282] INFO - BUILD INDEX on default(employeea8b0b7da499e49c39f9bb8566f466ea9job_title,employeea8b0b7da499e49c39f9bb8566f466ea9join_yr) USING GSI
[2022-09-11 03:49:06,747] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeea8b0b7da499e49c39f9bb8566f466ea9job_title,employeea8b0b7da499e49c39f9bb8566f466ea9join_yr) USING GSI
[2022-09-11 03:49:06,772] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeea8b0b7da499e49c39f9bb8566f466ea9job_title%2Cemployeea8b0b7da499e49c39f9bb8566f466ea9join_yr%29+USING+GSI
[2022-09-11 03:49:06,811] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.50016ms
[2022-09-11 03:49:07,845] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea8b0b7da499e49c39f9bb8566f466ea9job_title'
[2022-09-11 03:49:07,871] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8b0b7da499e49c39f9bb8566f466ea9job_title%27
[2022-09-11 03:49:07,882] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.689198ms
[2022-09-11 03:49:08,913] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea8b0b7da499e49c39f9bb8566f466ea9job_title'
[2022-09-11 03:49:08,938] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8b0b7da499e49c39f9bb8566f466ea9job_title%27
[2022-09-11 03:49:08,948] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.039271ms
[2022-09-11 03:49:08,973] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea8b0b7da499e49c39f9bb8566f466ea9join_yr'
[2022-09-11 03:49:08,999] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8b0b7da499e49c39f9bb8566f466ea9join_yr%27
[2022-09-11 03:49:09,002] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.573511ms
[2022-09-11 03:49:10,033] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea8b0b7da499e49c39f9bb8566f466ea9job_title'
[2022-09-11 03:49:10,060] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8b0b7da499e49c39f9bb8566f466ea9job_title%27
[2022-09-11 03:49:10,070] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.106965ms
[2022-09-11 03:49:10,097] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea8b0b7da499e49c39f9bb8566f466ea9job_title ON default USING GSI
[2022-09-11 03:49:10,123] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeea8b0b7da499e49c39f9bb8566f466ea9job_title+ON+default+USING+GSI
[2022-09-11 03:49:10,180] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.154766ms
[2022-09-11 03:49:10,216] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea8b0b7da499e49c39f9bb8566f466ea9join_yr'
[2022-09-11 03:49:10,242] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8b0b7da499e49c39f9bb8566f466ea9join_yr%27
[2022-09-11 03:49:10,253] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.314356ms
[2022-09-11 03:49:10,281] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea8b0b7da499e49c39f9bb8566f466ea9join_yr ON default USING GSI
[2022-09-11 03:49:10,307] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeea8b0b7da499e49c39f9bb8566f466ea9join_yr+ON+default+USING+GSI
[2022-09-11 03:49:10,342] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.768681ms
[2022-09-11 03:49:10,378] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea8b0b7da499e49c39f9bb8566f466ea9job_title'
[2022-09-11 03:49:10,403] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8b0b7da499e49c39f9bb8566f466ea9job_title%27
[2022-09-11 03:49:10,411] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.14286ms
[2022-09-11 03:49:10,412] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e8f9a5f1-abab-4764-be69-fc39ffe91178', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.14286ms', 'executionTime': '7.073513ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:49:10,440] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea8b0b7da499e49c39f9bb8566f466ea9join_yr'
[2022-09-11 03:49:10,465] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8b0b7da499e49c39f9bb8566f466ea9join_yr%27
[2022-09-11 03:49:10,468] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.561797ms
[2022-09-11 03:49:10,469] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1a914ebf-26ed-4dcb-9b5f-fe73953b27ff', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.561797ms', 'executionTime': '1.509506ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:49:10,573] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:49:10,576] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:10,576] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:11,169] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:11,224] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:49:11,225] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:49:11,324] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:49:11,382] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:49:11,386] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:11,386] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:11,986] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:12,041] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:49:12,042] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:49:12,139] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:49:12,194] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:49:12,194] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:49:12,194] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:49:12,221] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:49:12,247] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:49:12,258] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.958536ms
[2022-09-11 03:49:12,283] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:49:12,309] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:49:12,366] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.93675ms
[2022-09-11 03:49:12,430] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:49:12,430] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.12276285752256, 'mem_free': 12861554688, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:49:12,430] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:49:12,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:12,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:13,045] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:13,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:13,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:13,648] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:13,653] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:13,653] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:14,407] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:14,414] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:14,414] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:15,459] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:22,508] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-11 03:49:22,657] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:49:23,024] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:49:23,052] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:49:23,053] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:49:23,111] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:49:23,167] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:49:23,222] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:49:23,223] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:49:23,302] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:49:23,304] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:49:23,329] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:49:23,456] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:49:23,456] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:49:23,482] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:49:23,507] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:49:23,507] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:49:23,533] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:49:23,558] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:49:23,559] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:49:23,585] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:49:23,611] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:49:23,612] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:49:23,637] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:49:23,638] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-11 03:49:23,638] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:49:23,638] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 83.509s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_8'}
[2022-09-11 03:49:23,733] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:23,733] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:24,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:24,361] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:49:24,439] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:49:24,439] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-11 03:49:24,440] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:49:24,760] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:49:24,787] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 12997136384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:49:24,814] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:49:24,815] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:49:24,815] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:49:24,848] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:49:24,882] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:49:24,883] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:49:24,910] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:49:24,911] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:49:24,911] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:49:24,912] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:49:24,959] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:49:24,963] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:24,963] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:25,542] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:25,544] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:49:25,653] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:49:25,654] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:49:25,683] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:49:25,683] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:49:25,710] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:49:25,794] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:49:25,823] - [task:165] INFO -  {'uptime': '667', 'memoryTotal': 15466930176, 'memoryFree': 12996435968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:49:25,849] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:49:25,877] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:49:25,877] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:49:25,927] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:49:25,930] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:25,930] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:26,506] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:26,507] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:49:26,628] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:49:26,629] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:49:26,657] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:49:26,658] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:49:26,683] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:49:26,766] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:49:26,793] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 12988841984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:49:26,819] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:49:26,847] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:49:26,848] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:49:26,900] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:49:26,905] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:26,905] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:27,484] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:27,486] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:49:27,599] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:49:27,600] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:49:27,630] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:49:27,631] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:49:27,656] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:49:27,741] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:49:27,767] - [task:165] INFO -  {'uptime': '663', 'memoryTotal': 15466930176, 'memoryFree': 12874981376, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:49:27,795] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:49:27,823] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:49:27,823] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:49:27,875] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:49:27,879] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:27,879] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:28,473] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:28,474] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:49:28,592] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:49:28,593] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:49:28,623] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:49:28,623] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:49:28,649] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:49:28,710] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:49:29,222] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:49:34,227] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:49:34,313] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:49:34,318] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:34,319] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:34,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:34,923] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:49:35,035] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:49:35,037] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:49:35,037] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:49:35,845] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:49:35,900] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 03:49:35,900] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:49:36,802] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:49:37,110] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:49:37,384] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:49:37,387] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 03:49:37,442] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:37,442] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:38,063] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:38,068] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:38,068] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:38,918] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:38,929] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:38,930] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:39,977] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:39,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:39,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:41,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:47,255] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:49:47,256] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.25338307186005, 'mem_free': 12987830272, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:49:47,256] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:49:47,256] - [basetestcase:467] INFO - Time to execute basesetup : 23.525867462158203
[2022-09-11 03:49:47,308] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:49:47,308] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:49:47,360] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:49:47,361] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:49:47,414] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:49:47,414] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:49:47,468] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:49:47,468] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:49:47,519] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:49:47,578] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:49:47,578] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:49:47,579] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:49:52,591] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:49:52,594] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:49:52,594] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:49:53,222] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:49:54,268] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:49:54,550] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:49:56,676] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:49:56,761] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:49:56,762] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:49:56,762] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:50:26,789] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:50:26,819] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:50:26,844] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:50:26,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.449693ms
[2022-09-11 03:50:26,912] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f160fb4e-6d42-4055-b930-03062df44cf3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.449693ms', 'executionTime': '65.386013ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:50:26,912] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:50:26,938] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:50:26,962] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:50:27,658] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 694.084789ms
[2022-09-11 03:50:27,658] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:50:27,727] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:50:27,773] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:50:27,781] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.578531ms
[2022-09-11 03:50:28,003] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:50:28,040] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:50:28,063] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:50:28,064] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:50:28,082] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:50:28,083] - [base_gsi:326] INFO - []
[2022-09-11 03:50:28,958] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:50:28,984] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:50:29,038] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.4891ms
[2022-09-11 03:50:29,039] - [base_gsi:282] INFO - BUILD INDEX on default(employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr) USING GSI
[2022-09-11 03:50:30,069] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr) USING GSI
[2022-09-11 03:50:30,095] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr%29+USING+GSI
[2022-09-11 03:50:30,121] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.153647ms
[2022-09-11 03:50:31,153] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr'
[2022-09-11 03:50:31,178] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr%27
[2022-09-11 03:50:31,188] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.409866ms
[2022-09-11 03:50:32,223] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr'
[2022-09-11 03:50:32,249] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr%27
[2022-09-11 03:50:32,259] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.161888ms
[2022-09-11 03:50:33,289] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr'
[2022-09-11 03:50:33,315] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr%27
[2022-09-11 03:50:33,323] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.631481ms
[2022-09-11 03:50:33,348] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr ON default USING GSI
[2022-09-11 03:50:33,374] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr+ON+default+USING+GSI
[2022-09-11 03:50:33,418] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.846037ms
[2022-09-11 03:50:33,455] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr'
[2022-09-11 03:50:33,487] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed1ccd3287aa047548d700c1af19d57abjob_title_join_yr%27
[2022-09-11 03:50:33,497] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.396437ms
[2022-09-11 03:50:33,497] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5f092ea5-1607-47e8-9926-dc3a2772fd1e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.396437ms', 'executionTime': '8.337006ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:50:33,603] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:50:33,608] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:33,608] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:34,232] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:34,286] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:50:34,286] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:50:34,389] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:50:34,444] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:50:34,447] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:34,447] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:35,063] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:35,121] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:50:35,121] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:50:35,223] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:50:35,278] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:50:35,278] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:50:35,278] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:50:35,304] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:50:35,330] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:50:35,340] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.329766ms
[2022-09-11 03:50:35,365] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:50:35,391] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:50:35,432] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.449247ms
[2022-09-11 03:50:35,497] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:50:35,497] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.346372155405607, 'mem_free': 12843642880, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:50:35,498] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:50:35,502] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:35,502] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:36,113] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:36,118] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:36,118] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:37,097] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:37,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:37,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:38,191] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:38,202] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:38,202] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:39,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:45,771] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-11 03:50:45,925] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:50:46,508] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:50:46,536] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:50:46,536] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:50:46,590] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:50:46,644] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:50:46,697] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:50:46,698] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:50:46,777] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:50:46,778] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:50:46,803] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:50:46,936] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:50:46,937] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:50:46,964] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:50:46,989] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:50:46,990] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:50:47,016] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:50:47,046] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:50:47,047] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:50:47,079] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:50:47,105] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:50:47,105] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:50:47,131] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:50:47,132] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-11 03:50:47,132] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:50:47,133] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 83.459s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_9'}
[2022-09-11 03:50:47,226] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:47,227] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:47,827] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:47,860] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:50:47,940] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:50:47,940] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-11 03:50:47,940] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:50:48,253] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:50:48,280] - [task:165] INFO -  {'uptime': '752', 'memoryTotal': 15466930176, 'memoryFree': 12982095872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:50:48,308] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:50:48,308] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:50:48,308] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:50:48,337] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:50:48,374] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:50:48,374] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:50:48,402] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:50:48,403] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:50:48,403] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:50:48,403] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:50:48,451] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:50:48,454] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:48,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:49,063] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:49,064] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:50:49,186] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:50:49,187] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:50:49,217] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:50:49,218] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:50:49,243] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:50:49,329] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:50:49,356] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 12982525952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:50:49,381] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:50:49,410] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:50:49,411] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:50:49,461] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:50:49,464] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:49,464] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:50,097] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:50,098] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:50:50,212] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:50:50,214] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:50:50,243] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:50:50,243] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:50:50,269] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:50:50,353] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:50:50,379] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 12983287808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:50:50,404] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:50:50,432] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:50:50,433] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:50:50,482] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:50:50,487] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:50,487] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:51,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:51,087] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:50:51,209] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:50:51,210] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:50:51,240] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:50:51,241] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:50:51,267] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:50:51,351] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:50:51,378] - [task:165] INFO -  {'uptime': '748', 'memoryTotal': 15466930176, 'memoryFree': 12983312384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:50:51,403] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:50:51,431] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:50:51,432] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:50:51,483] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:50:51,486] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:51,486] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:52,138] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:52,139] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:50:52,266] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:50:52,268] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:50:52,299] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:50:52,300] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:50:52,327] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:50:52,387] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:50:53,050] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:50:58,055] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:50:58,138] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:50:58,146] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:50:58,146] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:50:58,757] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:50:58,758] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:50:58,872] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:50:58,874] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:50:58,874] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:50:59,522] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:50:59,581] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-11 03:50:59,582] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:51:00,482] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:51:00,801] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:51:01,089] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:51:01,093] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 03:51:01,150] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:51:01,150] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:51:01,812] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:51:01,817] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:51:01,817] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:51:02,803] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:51:02,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:51:02,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:51:03,915] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:51:03,924] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:51:03,924] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:51:05,000] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:51:11,419] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:51:11,419] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.23842807142156, 'mem_free': 12981956608, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:51:11,420] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:51:11,420] - [basetestcase:467] INFO - Time to execute basesetup : 24.196088314056396
[2022-09-11 03:51:11,472] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:51:11,472] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:51:11,526] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:51:11,527] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:51:11,580] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:51:11,580] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:51:11,634] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:51:11,634] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:51:11,687] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:51:11,749] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:51:11,750] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:51:11,750] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:51:16,759] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:51:16,764] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:51:16,764] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:51:17,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:51:18,421] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:51:18,704] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:51:20,793] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:51:20,872] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:51:20,873] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:51:20,873] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:51:50,899] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:51:50,929] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:51:50,955] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:51:51,025] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.089565ms
[2022-09-11 03:51:51,025] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ff5f1777-4d3b-4eb7-9d04-eef96144c491', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.089565ms', 'executionTime': '68.021727ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:51:51,025] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:51:51,051] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:51:51,077] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:51:51,733] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 654.654938ms
[2022-09-11 03:51:51,733] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:51:51,803] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:51:51,854] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:51:51,870] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 13.486712ms
[2022-09-11 03:51:52,112] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:51:52,154] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:51:52,177] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:51:52,177] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:51:52,187] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:51:52,247] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:51:53,071] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4ff058aab3024f979c174bce1f91bd89join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:51:53,097] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4ff058aab3024f979c174bce1f91bd89join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:51:53,150] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.183085ms
[2022-09-11 03:51:53,150] - [base_gsi:282] INFO - BUILD INDEX on default(`employee4ff058aab3024f979c174bce1f91bd89join_yr`) USING GSI
[2022-09-11 03:51:54,181] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee4ff058aab3024f979c174bce1f91bd89join_yr`) USING GSI
[2022-09-11 03:51:54,206] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee4ff058aab3024f979c174bce1f91bd89join_yr%60%29+USING+GSI
[2022-09-11 03:51:54,228] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.923926ms
[2022-09-11 03:51:54,268] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4ff058aab3024f979c174bce1f91bd89join_yr'
[2022-09-11 03:51:54,301] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4ff058aab3024f979c174bce1f91bd89join_yr%27
[2022-09-11 03:51:54,310] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.754122ms
[2022-09-11 03:51:55,341] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4ff058aab3024f979c174bce1f91bd89join_yr'
[2022-09-11 03:51:55,367] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4ff058aab3024f979c174bce1f91bd89join_yr%27
[2022-09-11 03:51:55,377] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.29652ms
[2022-09-11 03:51:56,408] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4ff058aab3024f979c174bce1f91bd89join_yr'
[2022-09-11 03:51:56,434] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4ff058aab3024f979c174bce1f91bd89join_yr%27
[2022-09-11 03:51:56,443] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.672348ms
[2022-09-11 03:51:57,474] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:51:57,500] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 03:51:57,505] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.098631ms
[2022-09-11 03:51:57,505] - [base_gsi:504] INFO - {'requestID': '01f83e90-08a9-45a3-a017-e9e1eec415e9', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4ff058aab3024f979c174bce1f91bd89join_yr', 'index_id': '56e9496feeb424f3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.098631ms', 'executionTime': '3.027826ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 03:51:57,506] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:51:57,506] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:51:57,506] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:51:57,506] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:51:57,507] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:51:57,508] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:51:57,508] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:51:57,508] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:51:57,567] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:51:57,606] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:51:57,631] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 03:51:57,811] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 168.223434ms
[2022-09-11 03:51:57,811] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:51:57,812] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:52:00,145] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:52:00,171] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4ff058aab3024f979c174bce1f91bd89join_yr'
[2022-09-11 03:52:00,197] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4ff058aab3024f979c174bce1f91bd89join_yr%27
[2022-09-11 03:52:00,203] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.421215ms
[2022-09-11 03:52:00,203] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '12b389d3-5d58-4cf8-8565-9a52f25c0df0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.421215ms', 'executionTime': '4.348549ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:52:00,307] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:52:00,310] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:00,311] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:01,059] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:01,115] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:52:01,116] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:52:01,222] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:52:01,281] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:52:01,285] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:01,285] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:01,976] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:02,034] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:52:02,034] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:52:02,147] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:52:02,208] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:52:02,209] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:52:02,209] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:52:02,234] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:52:02,263] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:52:02,268] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.557358ms
[2022-09-11 03:52:02,269] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '51e0a576-d59f-489b-9be3-014194c21cb6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.557358ms', 'executionTime': '3.496485ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:52:02,320] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:52:02,320] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.22470949069506, 'mem_free': 12843229184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:52:02,320] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:52:02,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:02,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:02,962] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:02,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:02,967] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:03,632] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:03,637] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:03,637] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:04,811] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:04,822] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:04,823] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:05,983] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:12,929] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-11 03:52:13,064] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:52:13,116] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:52:13,169] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:52:13,221] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:52:13,221] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:52:13,299] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:52:13,301] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:52:13,327] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:52:13,455] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:52:13,455] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:52:13,481] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:52:13,507] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:52:13,507] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:52:13,533] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:52:13,558] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:52:13,558] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:52:13,584] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:52:13,611] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:52:13,611] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:52:13,637] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:52:13,637] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-11 03:52:13,637] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:52:13,638] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.466s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_10'}
[2022-09-11 03:52:13,722] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:13,722] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:14,350] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:14,386] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:52:14,464] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:52:14,464] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-11 03:52:14,465] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:52:14,748] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:52:14,775] - [task:165] INFO -  {'uptime': '835', 'memoryTotal': 15466930176, 'memoryFree': 12987318272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:52:14,800] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:52:14,800] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:52:14,800] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:52:14,837] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:52:14,871] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:52:14,871] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:52:14,899] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:52:14,900] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:52:14,900] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:52:14,900] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:52:14,948] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:52:14,951] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:14,951] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:15,614] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:15,615] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:52:15,742] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:52:15,743] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:52:15,775] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:52:15,776] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:52:15,804] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:52:15,895] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:52:15,921] - [task:165] INFO -  {'uptime': '838', 'memoryTotal': 15466930176, 'memoryFree': 12982050816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:52:15,947] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:52:15,977] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:52:15,977] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:52:16,030] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:52:16,035] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:16,036] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:16,705] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:16,706] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:52:16,836] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:52:16,837] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:52:16,868] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:52:16,869] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:52:16,895] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:52:16,982] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:52:17,012] - [task:165] INFO -  {'uptime': '833', 'memoryTotal': 15466930176, 'memoryFree': 12848754688, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:52:17,040] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:52:17,070] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:52:17,070] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:52:17,124] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:52:17,128] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:17,128] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:17,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:17,773] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:52:17,892] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:52:17,893] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:52:17,923] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:52:17,923] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:52:17,949] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:52:18,034] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:52:18,062] - [task:165] INFO -  {'uptime': '834', 'memoryTotal': 15466930176, 'memoryFree': 12990042112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:52:18,088] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:52:18,116] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:52:18,116] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:52:18,170] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:52:18,174] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:18,174] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:18,802] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:18,803] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:52:18,927] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:52:18,928] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:52:18,958] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:52:18,959] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:52:18,984] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:52:19,041] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:52:19,599] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:52:24,604] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:52:24,694] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:52:24,699] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:24,699] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:25,368] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:25,370] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:52:25,488] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:52:25,489] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:52:25,490] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:52:26,167] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:52:26,220] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 03:52:26,221] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:52:27,118] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:52:27,427] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:52:27,685] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:52:27,688] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 03:52:27,742] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:27,742] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:28,371] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:28,376] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:28,376] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:29,257] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:29,265] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:29,265] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:30,412] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:30,426] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:30,426] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:31,534] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:38,319] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:52:38,319] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.68699475282938, 'mem_free': 12976656384, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:52:38,320] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:52:38,320] - [basetestcase:467] INFO - Time to execute basesetup : 24.600230932235718
[2022-09-11 03:52:38,372] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:52:38,372] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:52:38,424] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:52:38,424] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:52:38,477] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:52:38,478] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:52:38,532] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:52:38,532] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:52:38,583] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:52:38,644] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:52:38,645] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:52:38,645] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:52:43,655] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:52:43,658] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:52:43,658] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:52:44,276] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:52:45,335] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:52:45,504] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:52:47,703] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:52:47,785] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:52:47,786] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:52:47,786] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:53:17,801] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:53:17,831] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:53:17,856] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:53:17,924] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.042213ms
[2022-09-11 03:53:17,924] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '65101ea2-fe27-4331-88d6-f51003a86570', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.042213ms', 'executionTime': '64.963945ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:53:17,924] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:53:17,950] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:53:17,975] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:53:18,631] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 654.290626ms
[2022-09-11 03:53:18,631] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:53:18,686] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:53:18,720] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:53:18,729] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.595495ms
[2022-09-11 03:53:18,930] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:53:18,964] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:53:18,980] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:53:18,981] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:53:19,002] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:53:19,094] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:53:19,894] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee87e05e3a904a46debbe379ed92fbead8join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:53:19,919] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee87e05e3a904a46debbe379ed92fbead8join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:53:19,991] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.658335ms
[2022-09-11 03:53:19,992] - [base_gsi:282] INFO - BUILD INDEX on default(`employee87e05e3a904a46debbe379ed92fbead8join_yr`) USING GSI
[2022-09-11 03:53:21,022] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee87e05e3a904a46debbe379ed92fbead8join_yr`) USING GSI
[2022-09-11 03:53:21,049] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee87e05e3a904a46debbe379ed92fbead8join_yr%60%29+USING+GSI
[2022-09-11 03:53:21,074] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.834578ms
[2022-09-11 03:53:21,119] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87e05e3a904a46debbe379ed92fbead8join_yr'
[2022-09-11 03:53:21,150] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87e05e3a904a46debbe379ed92fbead8join_yr%27
[2022-09-11 03:53:21,160] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.798309ms
[2022-09-11 03:53:22,191] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87e05e3a904a46debbe379ed92fbead8join_yr'
[2022-09-11 03:53:22,218] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87e05e3a904a46debbe379ed92fbead8join_yr%27
[2022-09-11 03:53:22,228] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.592168ms
[2022-09-11 03:53:23,259] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87e05e3a904a46debbe379ed92fbead8join_yr'
[2022-09-11 03:53:23,286] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87e05e3a904a46debbe379ed92fbead8join_yr%27
[2022-09-11 03:53:23,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.760841ms
[2022-09-11 03:53:24,325] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:53:24,352] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 03:53:24,356] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.845058ms
[2022-09-11 03:53:24,356] - [base_gsi:504] INFO - {'requestID': 'e4d63864-c81d-4bed-a9bc-b6f2dda0a28d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee87e05e3a904a46debbe379ed92fbead8join_yr', 'index_id': 'fa4c5f0bbc3e631d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.845058ms', 'executionTime': '2.7648ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 03:53:24,357] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:53:24,358] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:53:24,358] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:53:24,358] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:53:24,358] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:53:24,359] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:53:24,359] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:53:24,359] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:53:24,417] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:53:24,456] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:53:24,481] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 03:53:24,627] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 134.054233ms
[2022-09-11 03:53:24,628] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:53:24,628] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:53:25,825] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-11 03:53:25,879] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-11 03:53:25,909] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:53:25,935] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 03:53:25,938] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.513193ms
[2022-09-11 03:53:25,938] - [base_gsi:504] INFO - {'requestID': '9dd79340-8bc7-4a61-be7f-8edc3272eb52', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee87e05e3a904a46debbe379ed92fbead8join_yr', 'index_id': 'fa4c5f0bbc3e631d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.513193ms', 'executionTime': '1.440232ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 03:53:25,939] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:53:25,939] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:53:25,939] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:53:25,939] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:53:25,940] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:53:25,940] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:53:25,940] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:53:25,940] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:53:25,993] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:53:26,031] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:53:26,057] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 03:53:26,179] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 111.988248ms
[2022-09-11 03:53:26,179] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:53:26,180] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:53:27,370] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee87e05e3a904a46debbe379ed92fbead8join_yr ON default USING GSI
[2022-09-11 03:53:27,396] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee87e05e3a904a46debbe379ed92fbead8join_yr+ON+default+USING+GSI
[2022-09-11 03:53:27,445] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.722729ms
[2022-09-11 03:53:27,478] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee87e05e3a904a46debbe379ed92fbead8join_yr'
[2022-09-11 03:53:27,505] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee87e05e3a904a46debbe379ed92fbead8join_yr%27
[2022-09-11 03:53:27,516] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.403607ms
[2022-09-11 03:53:27,516] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '93fa8df8-4a63-4159-a7b2-c244d9320aec', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.403607ms', 'executionTime': '6.337066ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:53:27,629] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:53:27,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:27,632] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:28,346] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:28,402] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:53:28,402] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:53:28,518] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:53:28,576] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:53:28,579] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:28,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:29,268] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:29,322] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:53:29,322] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:53:29,438] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:53:29,493] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:53:29,493] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:53:29,494] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:53:29,519] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:53:29,547] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:53:29,557] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.950237ms
[2022-09-11 03:53:29,587] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:53:29,613] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:53:29,662] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.621796ms
[2022-09-11 03:53:29,734] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:53:29,735] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.110494507670381, 'mem_free': 12834729984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:53:29,735] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:53:29,738] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:29,738] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:30,454] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:30,459] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:30,459] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:31,137] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:31,141] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:31,142] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:32,147] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:32,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:32,156] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:33,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:41,009] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-11 03:53:41,150] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:53:41,648] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:53:41,676] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:53:41,676] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:53:41,730] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:53:41,782] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:53:41,835] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:53:41,836] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:53:41,913] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:53:41,914] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:53:41,939] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:53:42,066] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:53:42,067] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:53:42,093] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:53:42,119] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:53:42,119] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:53:42,146] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:53:42,171] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:53:42,171] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:53:42,197] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:53:42,224] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:53:42,224] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:53:42,250] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:53:42,250] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-11 03:53:42,250] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:53:42,251] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 88.598s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_11'}
[2022-09-11 03:53:42,349] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:42,349] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:43,013] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:43,046] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:53:43,124] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:53:43,124] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-11 03:53:43,125] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:53:43,375] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:53:43,403] - [task:165] INFO -  {'uptime': '927', 'memoryTotal': 15466930176, 'memoryFree': 12970504192, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:53:43,429] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:53:43,430] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:53:43,430] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:53:43,463] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:53:43,496] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:53:43,497] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:53:43,525] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:53:43,526] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:53:43,526] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:53:43,526] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:53:43,574] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:53:43,577] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:43,577] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:44,219] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:44,220] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:53:44,342] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:53:44,343] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:53:44,373] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:53:44,374] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:53:44,399] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:53:44,482] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:53:44,509] - [task:165] INFO -  {'uptime': '923', 'memoryTotal': 15466930176, 'memoryFree': 12970487808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:53:44,534] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:53:44,561] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:53:44,562] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:53:44,611] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:53:44,614] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:44,614] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:45,315] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:45,316] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:53:45,448] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:53:45,449] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:53:45,479] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:53:45,480] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:53:45,510] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:53:45,595] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:53:45,621] - [task:165] INFO -  {'uptime': '924', 'memoryTotal': 15466930176, 'memoryFree': 12970491904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:53:45,646] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:53:45,676] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:53:45,676] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:53:45,726] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:53:45,731] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:45,731] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:46,477] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:46,478] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:53:46,602] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:53:46,603] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:53:46,632] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:53:46,633] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:53:46,660] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:53:46,745] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:53:46,772] - [task:165] INFO -  {'uptime': '924', 'memoryTotal': 15466930176, 'memoryFree': 12970876928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:53:46,798] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:53:46,826] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:53:46,826] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:53:46,876] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:53:46,879] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:46,879] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:47,534] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:47,535] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:53:47,663] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:53:47,664] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:53:47,695] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:53:47,696] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:53:47,721] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:53:47,778] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:53:48,297] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:53:53,299] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:53:53,386] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:53:53,392] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:53,392] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:54,088] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:54,089] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:53:54,217] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:53:54,218] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:53:54,218] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:53:54,892] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:53:54,946] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 03:53:54,947] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:53:55,858] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:53:56,189] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:53:56,474] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:53:56,478] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 03:53:56,537] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:56,538] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:57,206] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:57,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:57,211] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:58,238] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:58,246] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:58,246] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:53:59,460] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:53:59,475] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:53:59,475] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:54:00,707] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:54:07,816] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:54:07,817] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.12652106132607, 'mem_free': 12964925440, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:54:07,817] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:54:07,817] - [basetestcase:467] INFO - Time to execute basesetup : 25.470836877822876
[2022-09-11 03:54:07,869] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:54:07,870] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:54:07,923] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:54:07,924] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:54:07,975] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:54:07,975] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:54:08,027] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:54:08,028] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:54:08,078] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:54:08,136] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:54:08,136] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:54:08,137] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:54:13,148] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:54:13,152] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:54:13,152] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:54:13,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:54:14,890] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:54:15,189] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:54:18,270] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:54:18,352] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:54:18,352] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:54:18,352] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:54:48,377] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:54:48,406] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:54:48,434] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:54:48,502] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.085109ms
[2022-09-11 03:54:48,502] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c5dc7823-e2ca-4caa-b9f6-918fc32fed09', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.085109ms', 'executionTime': '66.007651ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:54:48,503] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:54:48,528] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:54:48,553] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:54:49,244] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 688.903093ms
[2022-09-11 03:54:49,244] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:54:49,315] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:54:49,362] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:54:49,370] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.285678ms
[2022-09-11 03:54:49,575] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:54:49,614] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:54:49,636] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:54:49,637] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:54:49,654] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:54:49,716] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:54:50,538] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeead6936fd45744ef82c2276566c9d123join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-11 03:54:50,562] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeead6936fd45744ef82c2276566c9d123join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-11 03:54:50,635] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.937092ms
[2022-09-11 03:54:50,636] - [base_gsi:282] INFO - BUILD INDEX on default(`employeeead6936fd45744ef82c2276566c9d123join_yr`) USING GSI
[2022-09-11 03:54:51,668] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeeead6936fd45744ef82c2276566c9d123join_yr`) USING GSI
[2022-09-11 03:54:51,694] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeeead6936fd45744ef82c2276566c9d123join_yr%60%29+USING+GSI
[2022-09-11 03:54:51,717] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.914387ms
[2022-09-11 03:54:51,766] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeead6936fd45744ef82c2276566c9d123join_yr'
[2022-09-11 03:54:51,796] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeead6936fd45744ef82c2276566c9d123join_yr%27
[2022-09-11 03:54:51,805] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.736744ms
[2022-09-11 03:54:52,835] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeead6936fd45744ef82c2276566c9d123join_yr'
[2022-09-11 03:54:52,860] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeead6936fd45744ef82c2276566c9d123join_yr%27
[2022-09-11 03:54:52,867] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.222274ms
[2022-09-11 03:54:53,899] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeead6936fd45744ef82c2276566c9d123join_yr'
[2022-09-11 03:54:53,924] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeead6936fd45744ef82c2276566c9d123join_yr%27
[2022-09-11 03:54:53,932] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.805038ms
[2022-09-11 03:54:54,963] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:54:54,990] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-11 03:54:54,994] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.807929ms
[2022-09-11 03:54:54,994] - [base_gsi:504] INFO - {'requestID': 'fb9dac3a-9105-4b41-a3df-ac8b5b00a218', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeead6936fd45744ef82c2276566c9d123join_yr', 'index_id': 'befc121022db14af', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.807929ms', 'executionTime': '2.734358ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-11 03:54:54,995] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-11 03:54:54,995] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:54:54,995] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-11 03:54:54,995] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-11 03:54:54,996] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:54:54,996] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:54:54,996] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:54:54,996] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-11 03:54:55,062] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-11 03:54:55,101] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-11 03:54:55,127] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-11 03:54:55,318] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 178.137404ms
[2022-09-11 03:54:55,319] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-11 03:54:55,319] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-11 03:54:57,463] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 03:55:00,015] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:55:00,080] - [on_prem_rest_client:3047] INFO - 0.07 seconds to create bucket default
[2022-09-11 03:55:00,081] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:55:00,977] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:55:01,293] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:55:01,573] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:55:01,576] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-11 03:55:03,609] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:55:03,609] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:03,637] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:55:03,666] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:55:03,666] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:03,693] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:55:03,722] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:55:03,722] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:03,750] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:55:03,779] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:55:03,779] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:03,806] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:55:03,860] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:55:03,887] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeead6936fd45744ef82c2276566c9d123join_yr'
[2022-09-11 03:55:03,913] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeead6936fd45744ef82c2276566c9d123join_yr%27
[2022-09-11 03:55:03,961] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.649626ms
[2022-09-11 03:55:03,961] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3a1b5982-f53e-4ceb-b91b-152a892f4010', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '45.649626ms', 'executionTime': '45.584939ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:55:03,987] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeead6936fd45744ef82c2276566c9d123join_yr'
[2022-09-11 03:55:04,012] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeead6936fd45744ef82c2276566c9d123join_yr%27
[2022-09-11 03:55:04,015] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.097455ms
[2022-09-11 03:55:04,015] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '414301f2-8c8c-4041-8e11-98aad9c46f7d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.097455ms', 'executionTime': '1.038643ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:55:04,065] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:55:04,193] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:55:04,195] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:04,195] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:04,894] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:04,951] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:55:04,952] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:55:05,072] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:55:05,128] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:55:05,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:05,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:05,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:05,899] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:55:05,900] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:55:06,016] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:55:06,074] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:55:06,074] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:55:06,075] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:55:06,101] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:55:06,127] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:55:06,136] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.319828ms
[2022-09-11 03:55:06,136] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b1d125e9-e3e7-4f07-8d8f-dee0cd88c010', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.319828ms', 'executionTime': '7.25081ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:55:06,136] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:55:06,162] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:55:06,188] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:55:06,190] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.101913ms
[2022-09-11 03:55:06,191] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1f951f7b-0886-420b-a8d3-1f17cc74a589', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.101913ms', 'executionTime': '1.036819ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:55:06,244] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:55:06,244] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.39590951672761, 'mem_free': 12785623040, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:55:06,244] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:55:06,248] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:06,248] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:06,925] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:06,930] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:06,930] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:07,718] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:07,726] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:07,726] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:08,968] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:08,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:08,976] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:10,237] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:17,656] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 03:55:17,795] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:55:18,099] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:55:18,127] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:55:18,127] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:55:18,181] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:55:18,237] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:55:18,290] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:55:18,291] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:55:18,369] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:55:18,370] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:55:18,396] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:18,527] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:55:18,527] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:18,554] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:55:18,580] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:55:18,580] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:18,606] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:55:18,633] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:55:18,633] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:18,660] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:55:18,685] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:55:18,685] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:18,711] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:55:18,712] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-11 03:55:18,712] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:55:18,713] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.417s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-11_03-38-11/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-11 03:55:18,770] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:18,770] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:19,504] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:19,539] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:55:19,620] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-11 03:55:19,621] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-11 03:55:19,621] - [basetestcase:224] INFO - initializing cluster
[2022-09-11 03:55:19,797] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-11 03:55:19,824] - [task:165] INFO -  {'uptime': '1023', 'memoryTotal': 15466930176, 'memoryFree': 12956241920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:55:19,849] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-11 03:55:19,850] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-11 03:55:19,850] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-11 03:55:19,881] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-11 03:55:19,921] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-11 03:55:19,922] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-11 03:55:19,950] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-11 03:55:19,951] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-11 03:55:19,951] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-11 03:55:19,952] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-11 03:55:19,999] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:55:20,002] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:20,002] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:20,694] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:20,696] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:55:20,821] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:55:20,822] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:55:20,852] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:55:20,852] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:55:20,878] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:55:20,965] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-11 03:55:20,992] - [task:165] INFO -  {'uptime': '1024', 'memoryTotal': 15466930176, 'memoryFree': 12955873280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:55:21,017] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:55:21,045] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-11 03:55:21,046] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-11 03:55:21,096] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:55:21,099] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:21,099] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:21,770] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:21,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:55:21,896] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:55:21,897] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:55:21,926] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:55:21,927] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:55:21,953] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:55:22,035] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-11 03:55:22,061] - [task:165] INFO -  {'uptime': '1019', 'memoryTotal': 15466930176, 'memoryFree': 12956553216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:55:22,087] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:55:22,121] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-11 03:55:22,121] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-11 03:55:22,170] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:55:22,173] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:22,173] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:22,856] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:22,857] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:55:22,990] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:55:22,992] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:55:23,026] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:55:23,027] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:55:23,056] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:55:23,148] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-11 03:55:23,175] - [task:165] INFO -  {'uptime': '1019', 'memoryTotal': 15466930176, 'memoryFree': 12956454912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-11 03:55:23,202] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-11 03:55:23,230] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-11 03:55:23,231] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-11 03:55:23,284] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-11 03:55:23,287] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:23,287] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:24,005] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:24,006] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:55:24,146] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:55:24,147] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:55:24,178] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-11 03:55:24,178] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-11 03:55:24,205] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-11 03:55:24,263] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-11 03:55:24,775] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:55:29,777] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-11 03:55:29,862] - [basetestcase:262] INFO - done initializing cluster
[2022-09-11 03:55:29,866] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:29,866] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:30,575] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:30,576] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-11 03:55:30,704] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-11 03:55:30,706] - [remote_util:5249] INFO - b'ok'
[2022-09-11 03:55:30,706] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-11 03:55:31,379] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-11 03:55:31,433] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-11 03:55:31,433] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-11 03:55:32,330] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:55:32,640] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:55:32,920] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-11 03:55:32,923] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-11 03:55:32,983] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:32,983] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:33,708] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:33,713] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:33,713] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:34,787] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:34,795] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:34,796] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:36,032] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:36,041] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:36,041] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:37,263] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:44,381] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:55:44,382] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.90227296155551, 'mem_free': 12955410432, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:55:44,382] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:55:44,382] - [basetestcase:467] INFO - Time to execute basesetup : 25.615519762039185
[2022-09-11 03:55:44,432] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:55:44,432] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:55:44,484] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:55:44,485] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:55:44,536] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:55:44,536] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:55:44,589] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-11 03:55:44,589] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-11 03:55:44,640] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:55:44,698] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-11 03:55:44,699] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-11 03:55:44,699] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-11 03:55:49,707] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-11 03:55:49,711] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:55:49,711] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:55:50,405] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:55:51,585] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-11 03:55:51,751] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-11 03:55:53,919] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-11 03:55:54,002] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:55:54,003] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-11 03:55:54,003] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-11 03:56:24,030] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-11 03:56:24,060] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:56:24,087] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:56:24,154] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.433049ms
[2022-09-11 03:56:24,154] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '326b73a1-2433-4ba0-b1be-384b6e9206ef', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.433049ms', 'executionTime': '65.37452ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-11 03:56:24,154] - [tuq_helper:758] INFO - Create primary index
[2022-09-11 03:56:24,181] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-11 03:56:24,207] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-11 03:56:24,900] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 691.209362ms
[2022-09-11 03:56:24,901] - [tuq_helper:760] INFO - Check if index is online
[2022-09-11 03:56:24,966] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:56:25,005] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:56:25,015] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.984392ms
[2022-09-11 03:56:25,218] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:56:25,266] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-11 03:56:25,284] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-11 03:56:25,285] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-11 03:56:25,300] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-11 03:56:25,365] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:56:25,500] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:56:25,503] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:56:25,503] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:56:26,287] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:56:26,341] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:56:26,342] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-11 03:56:26,467] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:56:26,525] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:56:26,528] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:56:26,528] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:56:27,266] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:56:27,322] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-11 03:56:27,322] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-11 03:56:27,446] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-11 03:56:27,502] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-11 03:56:27,503] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-11 03:56:27,503] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:56:27,528] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-11 03:56:27,552] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-11 03:56:27,561] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.275316ms
[2022-09-11 03:56:27,586] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-11 03:56:27,611] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-11 03:56:27,661] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.879472ms
[2022-09-11 03:56:27,735] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-11 03:56:27,736] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.848518374523818, 'mem_free': 12780929024, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-11 03:56:27,736] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-11 03:56:27,740] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:56:27,740] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:56:28,481] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:56:28,486] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:56:28,486] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:56:29,636] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:56:29,645] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:56:29,645] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:56:30,948] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:56:30,955] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-11 03:56:30,956] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-11 03:56:32,261] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-11 03:56:39,824] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-11 03:56:39,973] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-11 03:56:40,958] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-11 03:56:40,986] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-11 03:56:40,986] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-11 03:56:41,041] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:56:41,101] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:56:41,158] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-11 03:56:41,159] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-11 03:56:41,241] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-11 03:56:41,243] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-11 03:56:41,272] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:56:41,404] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-11 03:56:41,404] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:56:41,430] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-11 03:56:41,455] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-11 03:56:41,456] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:56:41,483] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-11 03:56:41,508] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-11 03:56:41,508] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:56:41,535] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-11 03:56:41,560] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-11 03:56:41,560] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-11 03:56:41,587] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-11 03:56:41,587] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-11 03:56:41,587] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-11 03:56:41,588] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 82.899s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 30025 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-11.09.2022-02.07.cfg
Build Log: make-11.09.2022-02.07.log
Server Log: logs-11.09.2022-02.07.tar.gz

Finished