Building

Started building at 2022/09/11 22:30:37
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 11.09.2022-22.30
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at 0760e59, changes since last good build: 
fatal: Invalid revision range 52396bd6b64d8340b403a6ca18517986458493d9..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 179393 at refs/changes/93/179393/3
Total 6 (delta 3), reused 5 (delta 3)
[unstable 9d6decec] MB-52109 Do not use round robin planner in serverless deployment
 Author: Deepkaran 
 Date: Mon Aug 15 13:50:46 2022 -0700
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179394 at refs/changes/94/179394/3
Total 11 (delta 6), reused 9 (delta 6)
[unstable 6bedfdc4] MB-52109 planner - allow single node subcluster during initial placement
 Author: Deepkaran 
 Date: Wed Aug 17 13:50:46 2022 -0700
 1 file changed, 9 insertions(+), 3 deletions(-)
Pulling in uncommitted change 179130 at refs/changes/30/179130/6
Total 20 (delta 11), reused 16 (delta 11)
[unstable 006239da] MB-52109 planner - add UsageThreshold to Plan
 Author: Deepkaran 
 Date: Fri Aug 19 13:50:46 2022 -0700
 3 files changed, 32 insertions(+), 14 deletions(-)
Pulling in uncommitted change 179131 at refs/changes/31/179131/6
Total 28 (delta 18), reused 24 (delta 18)
[unstable be302fd4] MB-52109 planner - add Units thresholds to config
 Author: Deepkaran 
 Date: Mon Aug 22 13:50:46 2022 -0700
 3 files changed, 30 insertions(+), 2 deletions(-)
Pulling in uncommitted change 179132 at refs/changes/32/179132/6
Total 34 (delta 21), reused 28 (delta 21)
[unstable 5e53e341] MB-52109 planner - use Units thresholds in LWM/HWM calculations
 Author: Deepkaran 
 Date: Wed Aug 24 13:50:46 2022 -0700
 2 files changed, 21 insertions(+), 6 deletions(-)
Pulling in uncommitted change 179128 at refs/changes/28/179128/13
Total 45 (delta 28), reused 36 (delta 28)
[unstable 85ef68bf] MB-52109 planner - tenant aware rebalance
 Author: Deepkaran 
 Date: Fri Aug 26 13:50:46 2022 -0700
 5 files changed, 707 insertions(+), 5 deletions(-)
Pulling in uncommitted change 179300 at refs/changes/00/179300/8
Total 51 (delta 32), reused 41 (delta 32)
[unstable 2a37594c] MB-52109 planner - implement Planner interface for TenantAwarePlanner
 Author: Deepkaran 
 Date: Mon Aug 29 13:50:46 2022 -0700
 2 files changed, 35 insertions(+), 4 deletions(-)
Pulling in uncommitted change 179395 at refs/changes/95/179395/7
Total 56 (delta 39), reused 49 (delta 39)
[unstable 999008e3] MB-52109 planner - add placement constraints for HWM/LWM usage
 Author: Deepkaran 
 Date: Wed Aug 31 13:50:46 2022 -0700
 1 file changed, 39 insertions(+), 22 deletions(-)
Pulling in uncommitted change 179185 at refs/changes/85/179185/13
Total 88 (delta 62), reused 80 (delta 62)
[unstable 5d4f6fee] MB-52109 tenant aware planner placement tests
 Author: Deepkaran 
 Date: Fri Sep 2 13:50:46 2022 -0700
 21 files changed, 1875 insertions(+), 11 deletions(-)
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_1.json
 create mode 100644 secondary/tests/testdata/planner/tenantaware/new_index_2.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/1_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_empty_nodes_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/2_non_empty_node_1_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_empty_nodes_2_sg.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/4_non_empty_nodes_2_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/6_non_empty_nodes_3_sg_e.json
Pulling in uncommitted change 179439 at refs/changes/39/179439/6
Total 119 (delta 84), reused 110 (delta 84)
[unstable 9eaca027] MB-52109 tenant aware planner rebalance tests
 Author: Deepkaran 
 Date: Tue Sep 6 13:50:46 2022 -0700
 20 files changed, 5493 insertions(+)
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_a_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_b_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_c_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_d_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_e_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/6_non_empty_nodes_3_sg_h_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_f_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_g_out.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h.json
 create mode 100755 secondary/tests/testdata/planner/tenantaware/topology/rebalance/8_non_empty_nodes_4_sg_h_out.json
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable 045f4197] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 2898155b] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 0c9da7e1] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable aa4bf33] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable a88c44f] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 035f520] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 4b03e0c] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 048edc1] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable b1da8b1] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable 5f01ad3] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable fd73348] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/11 23:24:40
Testing mode: sanity,unit,functional,integration
Using storage type: fdb
Setting ulimit to 200000

Simple Test

Sep 11 23:30:04 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 23:33:58 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 11 23:34:42 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 11 23:36:11 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 11 23:37:05 b" 'stop_on_failure': 'True'}"
Sep 11 23:37:05 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 23:37:05 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_1'}"
Sep 11 23:37:05 b'-->result: '
Sep 11 23:37:05 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 11 23:37:05 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 11 23:37:05 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_2'}"
Sep 11 23:37:05 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 23:37:05 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 11 23:37:05 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_3'}"
Sep 11 23:37:05 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 23:37:05 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 23:37:05 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 11 23:37:05 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_4'}"
Sep 11 23:37:05 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 23:37:05 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 23:37:05 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 23:46:50 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 11 23:47:34 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 11 23:51:20 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 23:55:15 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 23:57:56 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 11 23:57:56 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_5'}"
Sep 11 23:57:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 23:57:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 11 23:57:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 11 23:57:56 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_6'}"
Sep 11 23:57:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 23:57:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 23:57:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 11 23:57:56 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_7'}"
Sep 11 23:57:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 23:57:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 23:57:56 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 11 23:57:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 11 23:57:56 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-11_23-25-01/test_8'}"
Sep 11 23:57:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 11 23:57:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 11 23:57:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 11 23:57:56 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 11 23:57:56 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 11 23:57:57 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 23:57:57 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 11 23:57:57 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 11 23:57:57 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 11 23:57:57 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 11 23:57:57 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 11 23:57:57 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 11 23:57:57 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 2 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=forestdb
Printing gsi_type=forestdb
gsi_type=forestdb
In here
-p makefile=True,gsi_type=forestdb
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'forestdb',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-12 00:00:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:00:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:00:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:00:49 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 00:00:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 00:00:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 00:01:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:38 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-12 00:01:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '44', 'memoryTotal': 15466930176, 'memoryFree': 13551063040, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:01:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:01:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:01:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '44', 'memoryTotal': 15466930176, 'memoryFree': 13583634432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:01:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 00:01:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:01:40 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:01:40 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '44', 'memoryTotal': 15466930176, 'memoryFree': 13581217792, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:01:40 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 00:01:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:01:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:01:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '39', 'memoryTotal': 15466930176, 'memoryFree': 13585162240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:01:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:01:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:01:42 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:01:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:01:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:01:42 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:01:42 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 00:01:42 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 00:01:42 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-12 00:01:43 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:01:48 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 00:01:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 00:01:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-12 00:01:48 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 00:01:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 00:01:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 00:01:50 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.01649639032696, 'mem_free': 13450596352, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 63.88416051864624
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 00:01:53 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 00:01:59 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 00:01:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:01:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:01:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:00 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 00:02:00 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 00:02:02 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 00:02:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:02:02 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 00:02:02 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 00:02:32 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 00:02:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 00:02:32 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 00:02:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 129.633245ms
2022-09-12 00:02:32 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '7dad231e-ca5d-4790-a6f9-5a04f34e6461', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '129.633245ms', 'executionTime': '129.428499ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 00:02:32 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 00:02:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 00:02:32 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 733.739197ms
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 14.090843ms
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 00:02:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.430176ms
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 59.782328ms
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.906010293611136, 'mem_free': 13353340928, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 00:02:39 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 00:02:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 00:02:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 111.191s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-12 00:02:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.22744851609781, 'mem_free': 13331980288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 00:02:46 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 00:02:46 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-12 00:02:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '113', 'memoryTotal': 15466930176, 'memoryFree': 13331980288, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 00:02:48 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:02:48 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '109', 'memoryTotal': 15466930176, 'memoryFree': 13327597568, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:02:48 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:02:49 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '110', 'memoryTotal': 15466930176, 'memoryFree': 13327917056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:02:49 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '109', 'memoryTotal': 15466930176, 'memoryFree': 13327212544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:02:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:02:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:02:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 00:02:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 00:02:50 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 00:02:50 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 00:02:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 00:02:55 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 00:02:55 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-12 00:02:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 00:02:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 00:02:56 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 00:02:56 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 00:02:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 00:02:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-12 00:02:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 00:02:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 00:02:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 00:02:57 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 00:02:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 00:02:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:02:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.94152036140817, 'mem_free': 13309763584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.104209899902344
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 00:03:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 00:03:07 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 00:03:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:03:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:03:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:08 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 00:03:08 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 00:03:10 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 00:03:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:03:10 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 00:03:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 00:03:40 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 00:03:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 00:03:40 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 00:03:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 64.238772ms
2022-09-12 00:03:40 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'a9c04429-4957-4f4f-a802-5bf86fe2ee19', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.238772ms', 'executionTime': '64.117381ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 00:03:40 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 00:03:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 00:03:40 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 726.935535ms
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.32149ms
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 00:03:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:03:42 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeee6449d9778be4c6cbd7ffe682423140ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-12 00:03:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeee6449d9778be4c6cbd7ffe682423140ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-12 00:03:42 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 69.965231ms
2022-09-12 00:03:42 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeee6449d9778be4c6cbd7ffe682423140ajob_title) USING GSI
2022-09-12 00:03:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeee6449d9778be4c6cbd7ffe682423140ajob_title) USING GSI
2022-09-12 00:03:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeee6449d9778be4c6cbd7ffe682423140ajob_title%29+USING+GSI
2022-09-12 00:03:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 30.702046ms
2022-09-12 00:03:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeee6449d9778be4c6cbd7ffe682423140ajob_title'
2022-09-12 00:03:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee6449d9778be4c6cbd7ffe682423140ajob_title%27
2022-09-12 00:03:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.400105ms
2022-09-12 00:03:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeee6449d9778be4c6cbd7ffe682423140ajob_title'
2022-09-12 00:03:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee6449d9778be4c6cbd7ffe682423140ajob_title%27
2022-09-12 00:03:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.838697ms
2022-09-12 00:03:47 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 00:03:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 00:03:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-12 00:03:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.761584ms
2022-09-12 00:03:47 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '3d2c20cc-e8ed-44d1-b915-43f5951c93b2', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee6449d9778be4c6cbd7ffe682423140ajob_title', 'index_id': '8100afcdf7394a48', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.761584ms', 'executionTime': '3.688398ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-12 00:03:47 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 00:03:47 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 00:03:47 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 00:03:47 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-12 00:03:47 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-12 00:03:47 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-12 00:03:47 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-12 00:03:47 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-12 00:03:47 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-12 00:03:48 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 00:03:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 00:03:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-12 00:03:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 129.261692ms
2022-09-12 00:03:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-12 00:03:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-12 00:03:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 00:03:49 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeee6449d9778be4c6cbd7ffe682423140ajob_title'
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee6449d9778be4c6cbd7ffe682423140ajob_title%27
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.275189ms
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeee6449d9778be4c6cbd7ffe682423140ajob_title ON default USING GSI
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeee6449d9778be4c6cbd7ffe682423140ajob_title+ON+default+USING+GSI
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 43.309539ms
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeee6449d9778be4c6cbd7ffe682423140ajob_title'
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee6449d9778be4c6cbd7ffe682423140ajob_title%27
2022-09-12 00:03:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.641087ms
2022-09-12 00:03:50 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '93e961d0-55bb-4455-b652-4f74e4580a59', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.641087ms', 'executionTime': '5.521109ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:03:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.171272ms
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 48.489743ms
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.433273505624735, 'mem_free': 13261651968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:03:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:03:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:03:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:03:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 00:03:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 00:03:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 00:03:56 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 00:03:57 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 00:03:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 76.448s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_2'}
[2022-09-12 00:03:57,451] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:03:57,451] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:03:57,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:03:57,749] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:03:57,834] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:03:57,834] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-12 00:03:57,835] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:03:58,479] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:03:58,506] - [task:165] INFO -  {'uptime': '184', 'memoryTotal': 15466930176, 'memoryFree': 13216329728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:03:58,532] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:03:58,532] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:03:58,532] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:03:58,564] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:03:58,599] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:03:58,600] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:03:58,628] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:03:58,629] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:03:58,629] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:03:58,629] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:03:58,677] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:03:58,680] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:03:58,680] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:03:58,932] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:03:58,933] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:03:59,001] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:03:59,002] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:03:59,035] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:03:59,035] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:03:59,064] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:03:59,151] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:03:59,179] - [task:165] INFO -  {'uptime': '185', 'memoryTotal': 15466930176, 'memoryFree': 13248454656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:03:59,205] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:03:59,233] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:03:59,234] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:03:59,284] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:03:59,287] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:03:59,287] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:03:59,540] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:03:59,541] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:03:59,609] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:03:59,610] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:03:59,641] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:03:59,642] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:03:59,668] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:03:59,753] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:03:59,779] - [task:165] INFO -  {'uptime': '180', 'memoryTotal': 15466930176, 'memoryFree': 13249486848, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:03:59,804] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:03:59,832] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:03:59,832] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:03:59,883] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:03:59,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:03:59,888] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:00,155] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:00,156] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:04:00,226] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:04:00,227] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:04:00,257] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:04:00,258] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:04:00,284] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:04:00,370] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:04:00,397] - [task:165] INFO -  {'uptime': '179', 'memoryTotal': 15466930176, 'memoryFree': 13247295488, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:04:00,423] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:04:00,451] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:04:00,451] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:04:00,503] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:04:00,506] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:04:00,506] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:00,782] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:00,783] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:04:00,855] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:04:00,856] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:04:00,887] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:04:00,888] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:04:00,915] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:04:00,974] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:04:01,635] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:04:06,641] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:04:06,724] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:04:06,740] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:04:06,741] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:07,033] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:07,034] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:04:07,107] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:04:07,108] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:04:07,108] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:04:08,090] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:04:08,143] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 00:04:08,143] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:04:09,134] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:04:09,441] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:04:09,770] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:04:09,772] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:04:09,835] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:04:09,835] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:10,081] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:10,086] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:04:10,087] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:10,328] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:10,333] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:04:10,333] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:10,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:10,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:04:10,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:10,868] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:14,468] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:04:14,468] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.92843447667986, 'mem_free': 13235630080, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:04:14,468] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:04:14,469] - [basetestcase:467] INFO - Time to execute basesetup : 17.020312070846558
[2022-09-12 00:04:14,519] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:04:14,519] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:04:14,572] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:04:14,572] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:04:14,625] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:04:14,625] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:04:14,683] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:04:14,683] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:04:14,733] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:04:14,795] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:04:14,795] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:04:14,796] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:04:19,805] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:04:19,808] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:04:19,808] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:04:20,057] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:04:20,973] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:04:21,138] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:04:22,851] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:04:22,873] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 00:04:23,529] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:04:23,529] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:04:23,529] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:04:53,555] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:04:53,583] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:04:53,609] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:04:53,676] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.60315ms
[2022-09-12 00:04:53,677] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '64be1902-834d-43f8-8747-4865ba1826ba', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.60315ms', 'executionTime': '65.458097ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:04:53,677] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:04:53,702] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:04:53,726] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:04:54,475] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 746.821ms
[2022-09-12 00:04:54,475] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:04:54,551] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:04:54,589] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:04:54,597] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.286043ms
[2022-09-12 00:04:54,795] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:04:54,828] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:04:54,845] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:04:54,845] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:04:54,858] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:04:54,931] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:04:55,757] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee06b0090e69534613ac9cfcd7bf052c78job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:04:55,782] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee06b0090e69534613ac9cfcd7bf052c78job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:04:55,845] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.616427ms
[2022-09-12 00:04:55,846] - [base_gsi:282] INFO - BUILD INDEX on default(employee06b0090e69534613ac9cfcd7bf052c78job_title) USING GSI
[2022-09-12 00:04:56,875] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee06b0090e69534613ac9cfcd7bf052c78job_title) USING GSI
[2022-09-12 00:04:56,901] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee06b0090e69534613ac9cfcd7bf052c78job_title%29+USING+GSI
[2022-09-12 00:04:56,924] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.666951ms
[2022-09-12 00:04:57,955] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06b0090e69534613ac9cfcd7bf052c78job_title'
[2022-09-12 00:04:57,980] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06b0090e69534613ac9cfcd7bf052c78job_title%27
[2022-09-12 00:04:57,990] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.427341ms
[2022-09-12 00:04:59,020] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06b0090e69534613ac9cfcd7bf052c78job_title'
[2022-09-12 00:04:59,045] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06b0090e69534613ac9cfcd7bf052c78job_title%27
[2022-09-12 00:04:59,054] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.626413ms
[2022-09-12 00:04:59,507] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 00:04:59,671] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:05:00,521] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:05:01,056] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:05:01,090] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:05:01,117] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 00:05:01,121] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.315291ms
[2022-09-12 00:05:01,121] - [task:3246] INFO - {'requestID': 'af536dad-c7f8-473a-9baf-7e292b6ada8c', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee06b0090e69534613ac9cfcd7bf052c78job_title', 'index_id': '820baf38fbdddad4', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.315291ms', 'executionTime': '2.240652ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 00:05:01,121] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:05:01,121] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:05:01,122] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:05:01,122] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:05:01,123] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 00:05:01,123] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:05:01,123] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:05:01,123] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:05:01,123] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:05:02,122] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:05:02,154] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:05:02,183] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 00:05:02,320] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 127.862929ms
[2022-09-12 00:05:02,321] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:05:02,321] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:05:03,113] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:05:03,113] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:05:04,145] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06b0090e69534613ac9cfcd7bf052c78job_title'
[2022-09-12 00:05:04,172] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06b0090e69534613ac9cfcd7bf052c78job_title%27
[2022-09-12 00:05:04,183] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.767742ms
[2022-09-12 00:05:04,212] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee06b0090e69534613ac9cfcd7bf052c78job_title ON default USING GSI
[2022-09-12 00:05:04,238] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee06b0090e69534613ac9cfcd7bf052c78job_title+ON+default+USING+GSI
[2022-09-12 00:05:04,286] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.976858ms
[2022-09-12 00:05:04,330] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee06b0090e69534613ac9cfcd7bf052c78job_title'
[2022-09-12 00:05:04,359] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee06b0090e69534613ac9cfcd7bf052c78job_title%27
[2022-09-12 00:05:04,367] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.487545ms
[2022-09-12 00:05:04,367] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '884bb786-6208-4d07-9cb3-9b6d91f516db', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.487545ms', 'executionTime': '6.404287ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:05:04,481] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:05:04,484] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:04,485] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:04,857] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:04,914] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:05:04,915] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:05:04,974] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:05:05,039] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:05:05,042] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:05,042] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:05,390] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:05,448] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:05:05,448] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:05:05,509] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:05:05,565] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:05:05,565] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:05:05,566] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:05:05,592] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:05:05,618] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:05:05,627] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.347054ms
[2022-09-12 00:05:05,652] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:05:05,678] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:05:05,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.602541ms
[2022-09-12 00:05:05,805] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:05:05,806] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.16601736784774, 'mem_free': 13120466944, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:05:05,806] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:05:05,810] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:05,810] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:06,146] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:06,151] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:06,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:06,536] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:06,541] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:06,541] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:07,202] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:07,208] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:07,209] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:07,804] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:11,747] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:05:11,892] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:05:12,643] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:05:12,671] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:05:12,671] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:05:12,728] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:05:12,783] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:05:12,836] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:05:12,837] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:05:12,919] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:05:12,920] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:05:12,945] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:05:13,103] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:05:13,104] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:05:13,133] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:05:13,159] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:05:13,159] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:05:13,186] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:05:13,212] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:05:13,212] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:05:13,238] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:05:13,269] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:05:13,269] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:05:13,296] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:05:13,296] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:05:13,296] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:05:13,297] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 75.900s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_3'}
[2022-09-12 00:05:13,395] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:13,395] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:13,697] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:13,728] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:05:13,814] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:05:13,814] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-12 00:05:13,815] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:05:14,425] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:05:14,454] - [task:165] INFO -  {'uptime': '260', 'memoryTotal': 15466930176, 'memoryFree': 13186936832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:05:14,480] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:05:14,480] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:05:14,480] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:05:14,517] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:05:14,550] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:05:14,550] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:05:14,579] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:05:14,580] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:05:14,580] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:05:14,580] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:05:14,631] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:05:14,634] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:14,634] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:14,963] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:14,964] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:05:15,043] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:05:15,044] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:05:15,076] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:05:15,077] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:05:15,104] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:05:15,191] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:05:15,218] - [task:165] INFO -  {'uptime': '260', 'memoryTotal': 15466930176, 'memoryFree': 13176160256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:05:15,245] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:05:15,274] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:05:15,274] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:05:15,327] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:05:15,330] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:15,330] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:15,666] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:15,667] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:05:15,749] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:05:15,750] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:05:15,783] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:05:15,784] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:05:15,811] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:05:15,899] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:05:15,926] - [task:165] INFO -  {'uptime': '255', 'memoryTotal': 15466930176, 'memoryFree': 13131407360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:05:15,953] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:05:15,981] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:05:15,982] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:05:16,033] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:05:16,038] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:16,038] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:16,364] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:16,365] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:05:16,440] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:05:16,441] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:05:16,470] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:05:16,471] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:05:16,497] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:05:16,579] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:05:16,606] - [task:165] INFO -  {'uptime': '255', 'memoryTotal': 15466930176, 'memoryFree': 13130973184, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:05:16,633] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:05:16,661] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:05:16,662] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:05:16,717] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:05:16,720] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:16,721] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:17,030] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:17,031] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:05:17,106] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:05:17,107] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:05:17,139] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:05:17,139] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:05:17,165] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:05:17,223] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:05:17,851] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:05:22,856] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:05:22,942] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:05:22,948] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:22,949] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:23,293] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:23,294] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:05:23,368] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:05:23,369] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:05:23,369] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:05:24,346] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:05:24,405] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 00:05:24,406] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:05:25,314] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:05:25,617] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:05:25,882] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:05:25,885] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:05:25,941] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:25,941] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:26,264] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:26,270] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:26,270] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:26,597] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:26,602] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:26,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:26,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:26,963] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:26,964] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:27,515] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:31,561] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:05:31,561] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.8726976705402, 'mem_free': 13187715072, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:05:31,562] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:05:31,562] - [basetestcase:467] INFO - Time to execute basesetup : 18.169246673583984
[2022-09-12 00:05:31,613] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:05:31,614] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:05:31,666] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:05:31,666] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:05:31,718] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:05:31,718] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:05:31,774] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:05:31,775] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:05:31,826] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:05:31,893] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:05:31,894] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:05:31,894] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:05:36,905] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:05:36,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:05:36,911] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:05:37,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:05:38,240] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:05:38,416] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:05:40,964] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:05:40,980] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 00:05:41,826] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:05:41,826] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:05:41,827] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:06:11,844] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:06:11,873] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:06:11,897] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:06:11,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.947565ms
[2022-09-12 00:06:11,967] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e0738106-10f1-4041-969f-15c99074242c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.947565ms', 'executionTime': '66.832318ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:06:11,967] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:06:11,992] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:06:12,016] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:06:12,737] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 719.035784ms
[2022-09-12 00:06:12,737] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:06:12,810] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:06:12,841] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:06:12,849] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.733839ms
[2022-09-12 00:06:13,048] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:06:13,089] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:06:13,102] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:06:13,102] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:06:13,117] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:06:13,184] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:06:14,010] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee8722588dffac42d68f8be28f9cb80338job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:06:14,035] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee8722588dffac42d68f8be28f9cb80338job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:06:14,104] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.123542ms
[2022-09-12 00:06:14,105] - [base_gsi:282] INFO - BUILD INDEX on default(employee8722588dffac42d68f8be28f9cb80338job_title) USING GSI
[2022-09-12 00:06:15,135] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee8722588dffac42d68f8be28f9cb80338job_title) USING GSI
[2022-09-12 00:06:15,160] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee8722588dffac42d68f8be28f9cb80338job_title%29+USING+GSI
[2022-09-12 00:06:15,182] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.849105ms
[2022-09-12 00:06:16,214] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8722588dffac42d68f8be28f9cb80338job_title'
[2022-09-12 00:06:16,240] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8722588dffac42d68f8be28f9cb80338job_title%27
[2022-09-12 00:06:16,249] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.529342ms
[2022-09-12 00:06:17,280] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8722588dffac42d68f8be28f9cb80338job_title'
[2022-09-12 00:06:17,306] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8722588dffac42d68f8be28f9cb80338job_title%27
[2022-09-12 00:06:17,313] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.472855ms
[2022-09-12 00:06:17,687] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 00:06:17,941] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:06:18,624] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:06:19,316] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:06:19,350] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:06:19,377] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 00:06:19,381] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.195631ms
[2022-09-12 00:06:19,381] - [task:3246] INFO - {'requestID': '8e5f193f-4b93-479b-8bb4-cd83a363135d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee8722588dffac42d68f8be28f9cb80338job_title', 'index_id': 'e299ad04ee8ef678', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.195631ms', 'executionTime': '2.121321ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 00:06:19,382] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:06:19,382] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:06:19,382] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:06:19,383] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:06:19,383] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 00:06:19,383] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:06:19,384] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:06:19,384] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:06:19,384] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:06:20,384] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:06:20,414] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:06:20,439] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 00:06:20,556] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 107.712119ms
[2022-09-12 00:06:20,556] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:06:20,557] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:06:21,324] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:06:21,324] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:06:22,356] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8722588dffac42d68f8be28f9cb80338job_title'
[2022-09-12 00:06:22,381] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8722588dffac42d68f8be28f9cb80338job_title%27
[2022-09-12 00:06:22,391] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.406979ms
[2022-09-12 00:06:22,418] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee8722588dffac42d68f8be28f9cb80338job_title ON default USING GSI
[2022-09-12 00:06:22,443] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee8722588dffac42d68f8be28f9cb80338job_title+ON+default+USING+GSI
[2022-09-12 00:06:22,490] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.787975ms
[2022-09-12 00:06:22,525] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8722588dffac42d68f8be28f9cb80338job_title'
[2022-09-12 00:06:22,551] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8722588dffac42d68f8be28f9cb80338job_title%27
[2022-09-12 00:06:22,559] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.156155ms
[2022-09-12 00:06:22,560] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7594eb05-15a5-4552-a47f-f1c8248fd63b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.156155ms', 'executionTime': '6.073201ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:06:22,665] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:06:22,668] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:22,668] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:23,068] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:23,124] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:06:23,124] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:06:23,193] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:06:23,250] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:06:23,253] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:23,253] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:23,647] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:23,705] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:06:23,705] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:06:23,772] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:06:23,827] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:06:23,827] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:06:23,828] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:06:23,853] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:06:23,878] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:06:23,888] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.056821ms
[2022-09-12 00:06:23,914] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:06:23,939] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:06:24,008] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.889163ms
[2022-09-12 00:06:24,075] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:06:24,075] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.42130226265396, 'mem_free': 13064732672, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:06:24,075] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:06:24,079] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:24,079] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:24,493] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:24,499] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:24,499] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:25,104] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:25,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:25,116] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:25,805] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:25,814] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:25,814] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:26,486] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:30,590] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:06:30,731] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:06:31,835] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:06:31,875] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:06:31,875] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:06:31,930] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:06:31,981] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:06:32,038] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:06:32,039] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:06:32,117] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:06:32,118] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:06:32,143] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:06:32,270] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:06:32,270] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:06:32,296] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:06:32,321] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:06:32,322] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:06:32,349] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:06:32,374] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:06:32,375] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:06:32,401] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:06:32,426] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:06:32,427] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:06:32,453] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:06:32,453] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:06:32,454] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:06:32,454] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.113s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_4'}
[2022-09-12 00:06:32,552] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:32,552] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:32,913] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:32,943] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:06:33,029] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:06:33,029] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-12 00:06:33,030] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:06:33,580] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:06:33,610] - [task:165] INFO -  {'uptime': '339', 'memoryTotal': 15466930176, 'memoryFree': 13141397504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:06:33,637] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:06:33,637] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:06:33,638] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:06:33,672] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:06:33,709] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:06:33,709] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:06:33,737] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:06:33,737] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:06:33,738] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:06:33,738] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:06:33,785] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:06:33,788] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:33,789] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:34,139] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:34,140] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:06:34,220] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:06:34,222] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:06:34,251] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:06:34,251] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:06:34,276] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:06:34,361] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:06:34,386] - [task:165] INFO -  {'uptime': '335', 'memoryTotal': 15466930176, 'memoryFree': 13064056832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:06:34,412] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:06:34,439] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:06:34,440] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:06:34,490] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:06:34,492] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:34,493] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:34,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:34,854] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:06:34,937] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:06:34,939] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:06:34,967] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:06:34,968] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:06:34,993] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:06:35,075] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:06:35,102] - [task:165] INFO -  {'uptime': '336', 'memoryTotal': 15466930176, 'memoryFree': 13065334784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:06:35,127] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:06:35,154] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:06:35,155] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:06:35,206] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:06:35,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:35,210] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:35,560] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:35,561] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:06:35,643] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:06:35,644] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:06:35,673] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:06:35,674] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:06:35,700] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:06:35,780] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:06:35,808] - [task:165] INFO -  {'uptime': '335', 'memoryTotal': 15466930176, 'memoryFree': 13063196672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:06:35,833] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:06:35,862] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:06:35,863] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:06:35,913] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:06:35,916] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:35,916] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:36,267] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:36,268] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:06:36,350] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:06:36,351] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:06:36,380] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:06:36,381] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:06:36,406] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:06:36,462] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:06:36,985] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:06:41,986] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:06:42,073] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:06:42,077] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:42,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:42,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:42,456] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:06:42,538] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:06:42,539] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:06:42,540] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:06:43,606] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:06:43,659] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 00:06:43,660] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:06:44,534] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:06:44,851] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:06:45,132] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:06:45,135] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:06:45,197] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:45,197] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:45,623] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:45,628] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:45,628] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:46,011] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:46,016] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:46,016] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:46,602] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:46,612] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:46,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:47,265] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:51,662] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:06:51,662] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.2567250988852, 'mem_free': 13141557248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:06:51,662] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:06:51,663] - [basetestcase:467] INFO - Time to execute basesetup : 19.11344337463379
[2022-09-12 00:06:51,714] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:06:51,714] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:06:51,766] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:06:51,766] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:06:51,822] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:06:51,822] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:06:51,876] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:06:51,876] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:06:51,928] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:06:51,993] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:06:51,993] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:06:51,993] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:06:57,006] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:06:57,009] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:06:57,009] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:06:57,388] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:06:58,363] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:06:58,621] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:07:01,135] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:07:01,151] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 00:07:01,837] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:07:01,837] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:07:01,837] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:07:31,858] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:07:31,888] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:07:31,914] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:07:31,982] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.160276ms
[2022-09-12 00:07:31,982] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '45bd4b9e-c867-4d22-9f5b-4d1909ca48d8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.160276ms', 'executionTime': '66.089961ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:07:31,982] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:07:32,007] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:07:32,031] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:07:32,738] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 704.894239ms
[2022-09-12 00:07:32,738] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:07:32,821] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:07:32,860] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:07:32,868] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.975955ms
[2022-09-12 00:07:33,066] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:07:33,099] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:07:33,114] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:07:33,114] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:07:33,129] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:07:33,221] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:07:34,019] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee517d96acb6544da1a6914976adf5fa1cjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:07:34,045] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee517d96acb6544da1a6914976adf5fa1cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:07:34,096] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.400149ms
[2022-09-12 00:07:34,097] - [base_gsi:282] INFO - BUILD INDEX on default(employee517d96acb6544da1a6914976adf5fa1cjob_title) USING GSI
[2022-09-12 00:07:35,127] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee517d96acb6544da1a6914976adf5fa1cjob_title) USING GSI
[2022-09-12 00:07:35,152] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee517d96acb6544da1a6914976adf5fa1cjob_title%29+USING+GSI
[2022-09-12 00:07:35,176] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.828665ms
[2022-09-12 00:07:36,208] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee517d96acb6544da1a6914976adf5fa1cjob_title'
[2022-09-12 00:07:36,233] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee517d96acb6544da1a6914976adf5fa1cjob_title%27
[2022-09-12 00:07:36,243] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.273374ms
[2022-09-12 00:07:37,273] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee517d96acb6544da1a6914976adf5fa1cjob_title'
[2022-09-12 00:07:37,302] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee517d96acb6544da1a6914976adf5fa1cjob_title%27
[2022-09-12 00:07:37,309] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.557024ms
[2022-09-12 00:07:37,775] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 00:07:37,948] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:07:38,810] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:07:38,989] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-12 00:07:38,989] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:07:39,161] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-12 00:07:39,161] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-12 00:07:39,162] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (3032836841, 0, b'')
[2022-09-12 00:07:39,311] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:07:39,338] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:07:39,364] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 00:07:39,368] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.198876ms
[2022-09-12 00:07:39,368] - [task:3246] INFO - {'requestID': '42bbb976-3a1b-4346-aee9-9ab343eeab6b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee517d96acb6544da1a6914976adf5fa1cjob_title', 'index_id': 'fe1482b81c86029b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.198876ms', 'executionTime': '2.125336ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 00:07:39,368] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:07:39,368] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:07:39,369] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:07:39,369] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:07:39,370] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 00:07:39,370] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:07:39,370] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:07:39,370] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:07:39,371] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:07:40,370] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:07:40,401] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:07:40,428] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 00:07:40,557] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 120.580971ms
[2022-09-12 00:07:40,557] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:07:40,557] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:07:41,333] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:07:41,334] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:07:42,365] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee517d96acb6544da1a6914976adf5fa1cjob_title'
[2022-09-12 00:07:42,392] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee517d96acb6544da1a6914976adf5fa1cjob_title%27
[2022-09-12 00:07:42,401] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.228892ms
[2022-09-12 00:07:42,429] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee517d96acb6544da1a6914976adf5fa1cjob_title ON default USING GSI
[2022-09-12 00:07:42,457] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee517d96acb6544da1a6914976adf5fa1cjob_title+ON+default+USING+GSI
[2022-09-12 00:07:42,497] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.144789ms
[2022-09-12 00:07:42,533] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee517d96acb6544da1a6914976adf5fa1cjob_title'
[2022-09-12 00:07:42,563] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee517d96acb6544da1a6914976adf5fa1cjob_title%27
[2022-09-12 00:07:42,572] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.564269ms
[2022-09-12 00:07:42,572] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a7c8bc06-5011-4e93-865a-7fbc085d3f70', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.564269ms', 'executionTime': '7.490166ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:07:42,679] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:07:42,683] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:42,683] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:43,131] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:43,187] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:07:43,187] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:07:43,263] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:07:43,318] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:07:43,322] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:43,322] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:43,766] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:43,825] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:07:43,825] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:07:43,900] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:07:43,957] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:07:43,958] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:07:43,958] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:07:43,984] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:07:44,009] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:07:44,023] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 11.729485ms
[2022-09-12 00:07:44,049] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:07:44,074] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:07:44,114] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.260551ms
[2022-09-12 00:07:44,191] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:07:44,191] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.48150311691966, 'mem_free': 13010444288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:07:44,192] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:07:44,196] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:44,196] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:44,618] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:44,623] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:44,624] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:45,169] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:45,180] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:45,181] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:45,947] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:45,959] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:45,959] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:46,769] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:51,500] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:07:51,643] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:07:52,241] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:07:52,269] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:07:52,269] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:07:52,323] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:07:52,378] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:07:52,436] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:07:52,437] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:07:52,518] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:07:52,519] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:07:52,547] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:07:52,685] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:07:52,686] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:07:52,712] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:07:52,738] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:07:52,739] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:07:52,777] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:07:52,804] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:07:52,804] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:07:52,830] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:07:52,856] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:07:52,856] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:07:52,883] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:07:52,883] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:07:52,883] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:07:52,884] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.387s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_5'}
[2022-09-12 00:07:52,985] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:52,985] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:53,393] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:53,423] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:07:53,503] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:07:53,504] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-12 00:07:53,504] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:07:54,011] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:07:54,038] - [task:165] INFO -  {'uptime': '420', 'memoryTotal': 15466930176, 'memoryFree': 13104529408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:07:54,063] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:07:54,063] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:07:54,064] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:07:54,095] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:07:54,130] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:07:54,130] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:07:54,158] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:07:54,159] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:07:54,159] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:07:54,159] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:07:54,207] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:07:54,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:54,210] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:54,603] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:54,604] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:07:54,691] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:07:54,692] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:07:54,722] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:07:54,723] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:07:54,748] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:07:54,829] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:07:54,854] - [task:165] INFO -  {'uptime': '416', 'memoryTotal': 15466930176, 'memoryFree': 13008302080, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:07:54,880] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:07:54,908] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:07:54,908] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:07:54,959] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:07:54,962] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:54,962] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:55,379] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:55,381] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:07:55,473] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:07:55,474] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:07:55,504] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:07:55,504] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:07:55,529] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:07:55,615] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:07:55,641] - [task:165] INFO -  {'uptime': '416', 'memoryTotal': 15466930176, 'memoryFree': 13008535552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:07:55,667] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:07:55,694] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:07:55,695] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:07:55,746] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:07:55,750] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:55,750] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:56,155] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:56,156] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:07:56,244] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:07:56,245] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:07:56,274] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:07:56,275] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:07:56,300] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:07:56,383] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:07:56,410] - [task:165] INFO -  {'uptime': '415', 'memoryTotal': 15466930176, 'memoryFree': 12999860224, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:07:56,436] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:07:56,464] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:07:56,464] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:07:56,513] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:07:56,517] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:07:56,517] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:07:56,929] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:07:56,930] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:07:57,023] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:07:57,024] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:07:57,055] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:07:57,056] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:07:57,083] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:07:57,142] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:07:57,687] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:08:02,692] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:08:02,782] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:08:02,787] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:08:02,787] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:08:03,236] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:08:03,237] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:08:03,326] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:08:03,327] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:08:03,328] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:08:04,273] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:08:04,330] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 00:08:04,331] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:08:05,264] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:08:05,585] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:08:05,871] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:08:05,875] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:08:05,939] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:08:05,939] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:08:06,360] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:08:06,365] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:08:06,365] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:08:06,756] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:08:06,761] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:08:06,761] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:08:07,439] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:08:07,449] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:08:07,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:08:08,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:08:12,978] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:08:12,979] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.82779727735214, 'mem_free': 13110321152, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:08:12,979] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:08:12,979] - [basetestcase:467] INFO - Time to execute basesetup : 19.99703812599182
[2022-09-12 00:08:13,033] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:08:13,033] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:08:13,087] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:08:13,087] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:08:13,141] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:08:13,142] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:08:13,195] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:08:13,195] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:08:13,246] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:08:13,306] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:08:13,308] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:08:13,309] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:08:18,321] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:08:18,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:08:18,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:08:18,765] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:08:19,753] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:08:20,011] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:08:22,449] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:08:22,466] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 00:08:23,541] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:08:23,541] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:08:23,541] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:08:53,570] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:08:53,600] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:08:53,625] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:08:53,691] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.763285ms
[2022-09-12 00:08:53,691] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '78374b5a-8d91-49e3-a353-41869f9fbd65', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.763285ms', 'executionTime': '63.664769ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:08:53,691] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:08:53,717] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:08:53,741] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:08:54,456] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 712.356698ms
[2022-09-12 00:08:54,456] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:08:54,531] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:08:54,560] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:08:54,568] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.313695ms
[2022-09-12 00:08:54,758] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:08:54,800] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:08:54,813] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:08:54,813] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:08:54,829] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:08:54,901] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:08:55,721] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea6e3293e5d8e4d00a0e7ed701f408f89job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:08:55,746] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeea6e3293e5d8e4d00a0e7ed701f408f89job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:08:55,800] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.108802ms
[2022-09-12 00:08:55,800] - [base_gsi:282] INFO - BUILD INDEX on default(employeea6e3293e5d8e4d00a0e7ed701f408f89job_title) USING GSI
[2022-09-12 00:08:56,836] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeea6e3293e5d8e4d00a0e7ed701f408f89job_title) USING GSI
[2022-09-12 00:08:56,861] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeea6e3293e5d8e4d00a0e7ed701f408f89job_title%29+USING+GSI
[2022-09-12 00:08:56,879] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 16.791665ms
[2022-09-12 00:08:57,910] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6e3293e5d8e4d00a0e7ed701f408f89job_title'
[2022-09-12 00:08:57,936] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6e3293e5d8e4d00a0e7ed701f408f89job_title%27
[2022-09-12 00:08:57,946] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.531131ms
[2022-09-12 00:08:58,977] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6e3293e5d8e4d00a0e7ed701f408f89job_title'
[2022-09-12 00:08:59,005] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6e3293e5d8e4d00a0e7ed701f408f89job_title%27
[2022-09-12 00:08:59,016] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.153611ms
[2022-09-12 00:08:59,509] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:08:59,682] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:09:03,364] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:09:04,022] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:09:04,057] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:09:04,083] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 00:09:04,087] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.931974ms
[2022-09-12 00:09:04,088] - [task:3246] INFO - {'requestID': '2493aa76-e113-4ee9-a642-da2dbae15364', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea6e3293e5d8e4d00a0e7ed701f408f89job_title', 'index_id': '1bb51cbf3b35a96', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.931974ms', 'executionTime': '2.870195ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-09-12 00:09:04,088] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:09:04,088] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:09:04,089] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:09:04,089] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:09:04,089] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 00:09:04,090] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:09:04,090] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:09:04,090] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:09:04,090] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:09:05,090] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:09:05,119] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:09:05,145] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 00:09:05,316] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 155.563286ms
[2022-09-12 00:09:05,316] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:09:05,317] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:09:06,875] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:09:06,875] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:09:07,908] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6e3293e5d8e4d00a0e7ed701f408f89job_title'
[2022-09-12 00:09:07,935] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6e3293e5d8e4d00a0e7ed701f408f89job_title%27
[2022-09-12 00:09:07,946] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.676364ms
[2022-09-12 00:09:07,972] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea6e3293e5d8e4d00a0e7ed701f408f89job_title ON default USING GSI
[2022-09-12 00:09:07,997] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeea6e3293e5d8e4d00a0e7ed701f408f89job_title+ON+default+USING+GSI
[2022-09-12 00:09:08,045] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.565445ms
[2022-09-12 00:09:08,081] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6e3293e5d8e4d00a0e7ed701f408f89job_title'
[2022-09-12 00:09:08,108] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6e3293e5d8e4d00a0e7ed701f408f89job_title%27
[2022-09-12 00:09:08,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.745003ms
[2022-09-12 00:09:08,116] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ab4bc978-4093-49f4-8bc9-ce0cb7e114d4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.745003ms', 'executionTime': '5.686424ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:09:08,224] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:09:08,228] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:08,228] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:08,761] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:08,817] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:09:08,818] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:09:08,906] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:09:08,962] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:09:08,965] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:08,965] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:09,534] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:09,594] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:09:09,594] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:09:09,684] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:09:09,743] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:09:09,743] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:09:09,743] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:09:09,770] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:09:09,797] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:09:09,807] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.883254ms
[2022-09-12 00:09:09,833] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:09:09,859] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:09:09,914] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.067568ms
[2022-09-12 00:09:09,992] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:09:09,992] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.17230649921501, 'mem_free': 12950593536, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:09:09,993] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:09:09,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:09,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:10,511] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:10,516] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:10,516] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:11,006] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:11,010] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:11,011] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:11,815] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:11,833] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:11,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:12,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:18,893] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:09:19,033] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:09:19,811] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:09:19,838] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:09:19,838] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:09:19,896] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:09:19,949] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:09:20,003] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:09:20,004] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:09:20,082] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:09:20,083] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:09:20,109] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:09:20,251] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:09:20,251] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:09:20,277] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:09:20,304] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:09:20,304] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:09:20,330] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:09:20,355] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:09:20,356] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:09:20,388] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:09:20,414] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:09:20,415] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:09:20,440] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:09:20,441] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:09:20,441] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:09:20,441] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 87.510s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_6'}
[2022-09-12 00:09:20,546] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:20,546] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:21,027] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:21,061] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:09:21,141] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:09:21,141] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-12 00:09:21,142] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:09:21,572] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:09:21,599] - [task:165] INFO -  {'uptime': '507', 'memoryTotal': 15466930176, 'memoryFree': 13034598400, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:09:21,625] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:09:21,625] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:09:21,626] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:09:21,654] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:09:21,688] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:09:21,688] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:09:21,716] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:09:21,717] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:09:21,717] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:09:21,717] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:09:21,765] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:09:21,768] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:21,768] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:22,242] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:22,243] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:09:22,340] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:09:22,342] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:09:22,372] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:09:22,372] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:09:22,397] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:09:22,479] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:09:22,506] - [task:165] INFO -  {'uptime': '506', 'memoryTotal': 15466930176, 'memoryFree': 13033164800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:09:22,531] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:09:22,559] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:09:22,559] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:09:22,611] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:09:22,614] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:22,614] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:23,113] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:23,115] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:09:23,216] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:09:23,217] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:09:23,246] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:09:23,246] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:09:23,271] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:09:23,354] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:09:23,380] - [task:165] INFO -  {'uptime': '506', 'memoryTotal': 15466930176, 'memoryFree': 13039046656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:09:23,405] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:09:23,433] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:09:23,433] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:09:23,484] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:09:23,488] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:23,488] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:24,004] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:24,005] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:09:24,109] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:09:24,111] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:09:24,146] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:09:24,147] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:09:24,173] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:09:24,255] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:09:24,281] - [task:165] INFO -  {'uptime': '506', 'memoryTotal': 15466930176, 'memoryFree': 13040234496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:09:24,306] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:09:24,333] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:09:24,334] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:09:24,383] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:09:24,386] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:24,386] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:24,874] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:24,875] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:09:24,974] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:09:24,975] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:09:25,005] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:09:25,005] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:09:25,030] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:09:25,085] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:09:25,725] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:09:30,731] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:09:30,818] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:09:30,823] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:30,823] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:31,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:31,370] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:09:31,476] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:09:31,477] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:09:31,477] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:09:32,234] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:09:32,289] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 00:09:32,290] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:09:33,197] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:09:33,502] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:09:33,780] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:09:33,784] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:09:33,843] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:33,843] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:34,373] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:34,377] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:34,377] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:34,968] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:34,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:34,976] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:35,841] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:35,849] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:35,850] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:36,763] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:42,250] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:09:42,250] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.69124023781443, 'mem_free': 13038800896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:09:42,251] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:09:42,251] - [basetestcase:467] INFO - Time to execute basesetup : 21.70774245262146
[2022-09-12 00:09:42,306] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:09:42,306] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:09:42,361] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:09:42,361] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:09:42,418] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:09:42,418] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:09:42,474] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:09:42,474] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:09:42,528] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:09:42,604] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:09:42,605] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:09:42,605] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:09:47,617] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:09:47,620] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:09:47,620] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:09:48,147] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:09:49,150] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:09:49,413] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:09:51,640] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:09:51,657] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 00:09:53,055] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:09:53,055] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:09:53,055] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:10:23,084] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:10:23,115] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:10:23,143] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:10:23,212] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.183308ms
[2022-09-12 00:10:23,212] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b01e04fa-9c33-47db-999a-6a8b3d9956b2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.183308ms', 'executionTime': '67.121503ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:10:23,212] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:10:23,240] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:10:23,268] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:10:23,908] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 638.173964ms
[2022-09-12 00:10:23,908] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:10:23,984] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:10:24,023] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:10:24,030] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.588857ms
[2022-09-12 00:10:24,238] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:10:24,273] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:10:24,292] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:10:24,292] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:10:24,306] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:10:24,377] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:10:25,189] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5808b0b2aeb84a0ea73c60945ab480dejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:10:25,214] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5808b0b2aeb84a0ea73c60945ab480dejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:10:25,282] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.382896ms
[2022-09-12 00:10:25,283] - [base_gsi:282] INFO - BUILD INDEX on default(employee5808b0b2aeb84a0ea73c60945ab480dejob_title) USING GSI
[2022-09-12 00:10:26,313] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5808b0b2aeb84a0ea73c60945ab480dejob_title) USING GSI
[2022-09-12 00:10:26,339] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee5808b0b2aeb84a0ea73c60945ab480dejob_title%29+USING+GSI
[2022-09-12 00:10:26,360] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.794623ms
[2022-09-12 00:10:27,392] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5808b0b2aeb84a0ea73c60945ab480dejob_title'
[2022-09-12 00:10:27,420] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5808b0b2aeb84a0ea73c60945ab480dejob_title%27
[2022-09-12 00:10:27,428] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.306584ms
[2022-09-12 00:10:28,457] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5808b0b2aeb84a0ea73c60945ab480dejob_title'
[2022-09-12 00:10:28,482] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5808b0b2aeb84a0ea73c60945ab480dejob_title%27
[2022-09-12 00:10:28,489] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.707606ms
[2022-09-12 00:10:28,976] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 00:10:29,147] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:10:30,309] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:10:30,917] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 00:10:31,092] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:10:32,400] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:10:32,795] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:10:32,974] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:10:36,648] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:10:37,509] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:10:37,541] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:10:37,566] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 00:10:37,571] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.409321ms
[2022-09-12 00:10:37,572] - [task:3246] INFO - {'requestID': '7d62db28-5eed-4db6-bc33-40679e7e103b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5808b0b2aeb84a0ea73c60945ab480dejob_title', 'index_id': 'afb06637981d15c7', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.409321ms', 'executionTime': '3.347858ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 00:10:37,572] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:10:37,572] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:10:37,573] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:10:37,573] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:10:37,573] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 00:10:37,574] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:10:37,574] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:10:37,574] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:10:37,574] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 00:10:38,574] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:10:38,604] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 00:10:38,630] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 00:10:39,004] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 153.081837ms
[2022-09-12 00:10:39,004] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:10:39,005] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:10:40,588] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:10:40,588] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 00:10:41,620] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5808b0b2aeb84a0ea73c60945ab480dejob_title'
[2022-09-12 00:10:41,646] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5808b0b2aeb84a0ea73c60945ab480dejob_title%27
[2022-09-12 00:10:41,656] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.891376ms
[2022-09-12 00:10:41,683] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5808b0b2aeb84a0ea73c60945ab480dejob_title ON default USING GSI
[2022-09-12 00:10:41,709] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5808b0b2aeb84a0ea73c60945ab480dejob_title+ON+default+USING+GSI
[2022-09-12 00:10:41,747] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.736012ms
[2022-09-12 00:10:41,786] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5808b0b2aeb84a0ea73c60945ab480dejob_title'
[2022-09-12 00:10:41,818] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5808b0b2aeb84a0ea73c60945ab480dejob_title%27
[2022-09-12 00:10:41,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.809843ms
[2022-09-12 00:10:41,827] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0c7dd6a9-f9f1-49a9-aa8b-73ca404fde2f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.809843ms', 'executionTime': '6.748817ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:10:41,944] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:10:41,947] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:41,947] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:42,671] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:42,727] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:10:42,728] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:10:42,835] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:10:42,895] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:10:42,898] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:42,898] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:43,579] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:43,637] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:10:43,637] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:10:43,741] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:10:43,797] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:10:43,797] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:10:43,798] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:10:43,823] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:10:43,850] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:10:43,860] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.754353ms
[2022-09-12 00:10:43,888] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:10:43,913] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:10:43,969] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.477144ms
[2022-09-12 00:10:44,046] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:10:44,046] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.35759546675832, 'mem_free': 12860563456, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:10:44,046] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:10:44,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:44,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:44,717] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:44,722] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:44,722] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:45,372] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:45,376] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:45,376] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:46,335] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:46,344] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:46,344] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:47,506] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:54,900] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:10:55,046] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:10:55,785] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:10:55,817] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:10:55,817] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:10:55,873] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:10:55,931] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:10:55,988] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:10:55,989] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:10:56,074] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:10:56,075] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:10:56,101] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:10:56,255] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:10:56,255] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:10:56,283] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:10:56,311] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:10:56,312] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:10:56,340] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:10:56,366] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:10:56,366] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:10:56,394] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:10:56,420] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:10:56,420] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:10:56,449] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:10:56,449] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 00:10:56,449] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:10:56,450] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 95.961s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_7'}
[2022-09-12 00:10:56,584] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:56,585] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:57,178] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:57,212] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:10:57,291] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:10:57,291] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-12 00:10:57,291] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:10:57,611] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:10:57,639] - [task:165] INFO -  {'uptime': '603', 'memoryTotal': 15466930176, 'memoryFree': 12973654016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:10:57,664] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:10:57,664] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:10:57,665] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:10:57,700] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:10:57,733] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:10:57,733] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:10:57,761] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:10:57,762] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:10:57,762] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:10:57,762] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:10:57,809] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:10:57,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:57,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:58,407] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:58,408] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:10:58,525] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:10:58,526] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:10:58,555] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:10:58,555] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:10:58,582] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:10:58,667] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:10:58,694] - [task:165] INFO -  {'uptime': '601', 'memoryTotal': 15466930176, 'memoryFree': 12973957120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:10:58,721] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:10:58,749] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:10:58,750] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:10:58,802] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:10:58,805] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:58,805] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:10:59,392] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:10:59,393] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:10:59,509] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:10:59,511] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:10:59,543] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:10:59,544] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:10:59,572] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:10:59,658] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:10:59,689] - [task:165] INFO -  {'uptime': '602', 'memoryTotal': 15466930176, 'memoryFree': 12974153728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:10:59,716] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:10:59,747] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:10:59,748] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:10:59,800] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:10:59,804] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:10:59,804] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:00,439] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:00,440] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:11:00,560] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:11:00,561] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:11:00,592] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:11:00,592] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:11:00,620] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:11:00,713] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:11:00,742] - [task:165] INFO -  {'uptime': '601', 'memoryTotal': 15466930176, 'memoryFree': 12973563904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:11:00,770] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:11:00,811] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:11:00,812] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:11:00,867] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:11:00,870] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:11:00,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:01,524] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:01,525] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:11:01,649] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:11:01,650] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:11:01,682] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:11:01,683] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:11:01,710] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:11:01,771] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:11:02,313] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:11:07,319] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:11:07,405] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:11:07,409] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:11:07,409] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:08,025] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:08,027] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:11:08,146] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:11:08,147] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:11:08,148] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:11:08,945] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:11:09,002] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 00:11:09,003] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:11:09,904] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:11:10,210] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:11:10,468] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:11:10,471] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 00:11:10,525] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:11:10,525] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:11,132] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:11,138] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:11:11,138] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:11,982] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:11,995] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:11:11,995] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:13,041] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:13,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:11:13,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:14,102] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:20,535] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:11:20,535] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.39813378304665, 'mem_free': 12986445824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:11:20,536] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:11:20,536] - [basetestcase:467] INFO - Time to execute basesetup : 23.95411467552185
[2022-09-12 00:11:20,591] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:11:20,591] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:11:20,651] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:11:20,652] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:11:20,709] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:11:20,709] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:11:20,766] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:11:20,767] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:11:20,822] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:11:20,887] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:11:20,888] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:11:20,888] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:11:25,900] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:11:25,903] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:11:25,903] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:11:26,506] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:11:27,538] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:11:27,713] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:11:29,947] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:11:30,035] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:11:30,035] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:11:30,035] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:12:00,055] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:12:00,086] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:12:00,112] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:12:00,181] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.148859ms
[2022-09-12 00:12:00,182] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9cab8cc7-80c2-4eb8-9f8f-e9ab564113e0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.148859ms', 'executionTime': '67.075641ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:12:00,182] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:12:00,210] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:12:00,235] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:12:00,932] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 694.240094ms
[2022-09-12 00:12:00,932] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:12:00,982] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:12:01,019] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:12:01,029] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.199607ms
[2022-09-12 00:12:01,249] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:12:01,283] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:12:01,303] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:12:01,306] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:12:01,321] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:12:01,324] - [base_gsi:326] INFO - []
[2022-09-12 00:12:02,207] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec5795fc095c948519a02153ed92678cbjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:12:02,233] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec5795fc095c948519a02153ed92678cbjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:12:02,310] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 74.400967ms
[2022-09-12 00:12:02,349] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec5795fc095c948519a02153ed92678cbjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:12:02,377] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec5795fc095c948519a02153ed92678cbjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:12:02,440] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.600289ms
[2022-09-12 00:12:02,440] - [base_gsi:282] INFO - BUILD INDEX on default(employeec5795fc095c948519a02153ed92678cbjob_title,employeec5795fc095c948519a02153ed92678cbjoin_yr) USING GSI
[2022-09-12 00:12:03,471] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec5795fc095c948519a02153ed92678cbjob_title,employeec5795fc095c948519a02153ed92678cbjoin_yr) USING GSI
[2022-09-12 00:12:03,496] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec5795fc095c948519a02153ed92678cbjob_title%2Cemployeec5795fc095c948519a02153ed92678cbjoin_yr%29+USING+GSI
[2022-09-12 00:12:03,534] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.658126ms
[2022-09-12 00:12:04,567] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec5795fc095c948519a02153ed92678cbjob_title'
[2022-09-12 00:12:04,594] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec5795fc095c948519a02153ed92678cbjob_title%27
[2022-09-12 00:12:04,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.496777ms
[2022-09-12 00:12:05,636] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec5795fc095c948519a02153ed92678cbjob_title'
[2022-09-12 00:12:05,663] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec5795fc095c948519a02153ed92678cbjob_title%27
[2022-09-12 00:12:05,672] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.586414ms
[2022-09-12 00:12:05,701] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec5795fc095c948519a02153ed92678cbjoin_yr'
[2022-09-12 00:12:05,728] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec5795fc095c948519a02153ed92678cbjoin_yr%27
[2022-09-12 00:12:05,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.532341ms
[2022-09-12 00:12:06,762] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec5795fc095c948519a02153ed92678cbjob_title'
[2022-09-12 00:12:06,789] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec5795fc095c948519a02153ed92678cbjob_title%27
[2022-09-12 00:12:06,799] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.034242ms
[2022-09-12 00:12:06,827] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec5795fc095c948519a02153ed92678cbjob_title ON default USING GSI
[2022-09-12 00:12:06,852] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec5795fc095c948519a02153ed92678cbjob_title+ON+default+USING+GSI
[2022-09-12 00:12:06,902] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.634307ms
[2022-09-12 00:12:06,947] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec5795fc095c948519a02153ed92678cbjoin_yr'
[2022-09-12 00:12:06,973] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec5795fc095c948519a02153ed92678cbjoin_yr%27
[2022-09-12 00:12:06,980] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.921089ms
[2022-09-12 00:12:07,007] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec5795fc095c948519a02153ed92678cbjoin_yr ON default USING GSI
[2022-09-12 00:12:07,032] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec5795fc095c948519a02153ed92678cbjoin_yr+ON+default+USING+GSI
[2022-09-12 00:12:07,078] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.779762ms
[2022-09-12 00:12:07,115] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec5795fc095c948519a02153ed92678cbjob_title'
[2022-09-12 00:12:07,154] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec5795fc095c948519a02153ed92678cbjob_title%27
[2022-09-12 00:12:07,161] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.814735ms
[2022-09-12 00:12:07,161] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a80f85d5-de3a-4054-ba7c-6f7af7677e48', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.814735ms', 'executionTime': '5.751151ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:12:07,189] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec5795fc095c948519a02153ed92678cbjoin_yr'
[2022-09-12 00:12:07,218] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec5795fc095c948519a02153ed92678cbjoin_yr%27
[2022-09-12 00:12:07,222] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.615499ms
[2022-09-12 00:12:07,222] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '037a11f3-2b24-4adc-80cf-e06530a6dc96', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.615499ms', 'executionTime': '1.535032ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:12:07,331] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:12:07,334] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:07,334] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:07,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:08,028] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:12:08,028] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:12:08,129] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:12:08,189] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:12:08,192] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:08,192] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:08,808] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:08,865] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:12:08,865] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:12:08,963] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:12:09,019] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:12:09,020] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:12:09,020] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:12:09,048] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:12:09,075] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:12:09,084] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.266673ms
[2022-09-12 00:12:09,110] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:12:09,135] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:12:09,185] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.809029ms
[2022-09-12 00:12:09,256] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:12:09,256] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.785338617687385, 'mem_free': 12871548928, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:12:09,257] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:12:09,262] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:09,262] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:09,912] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:09,917] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:09,918] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:10,543] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:10,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:10,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:11,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:11,545] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:11,545] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:12,587] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:19,368] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-12 00:12:19,514] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:12:20,519] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:12:20,546] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:12:20,547] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:12:20,602] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:12:20,670] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:12:20,725] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:12:20,726] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:12:20,806] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:12:20,807] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:12:20,833] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:12:20,968] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:12:20,968] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:12:20,997] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:12:21,024] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:12:21,025] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:12:21,051] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:12:21,077] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:12:21,077] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:12:21,104] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:12:21,129] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:12:21,129] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:12:21,162] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:12:21,162] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 00:12:21,162] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:12:21,163] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.638s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_8'}
[2022-09-12 00:12:21,247] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:21,247] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:21,861] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:21,896] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:12:21,977] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:12:21,977] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-12 00:12:21,978] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:12:22,274] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:12:22,302] - [task:165] INFO -  {'uptime': '688', 'memoryTotal': 15466930176, 'memoryFree': 12993466368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:12:22,330] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:12:22,330] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:12:22,330] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:12:22,366] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:12:22,403] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:12:22,403] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:12:22,433] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:12:22,434] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:12:22,434] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:12:22,434] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:12:22,482] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:12:22,485] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:22,485] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:23,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:23,078] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:12:23,189] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:12:23,190] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:12:23,221] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:12:23,222] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:12:23,247] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:12:23,331] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:12:23,357] - [task:165] INFO -  {'uptime': '687', 'memoryTotal': 15466930176, 'memoryFree': 12993642496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:12:23,383] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:12:23,410] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:12:23,410] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:12:23,460] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:12:23,463] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:23,464] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:24,058] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:24,059] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:12:24,175] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:12:24,176] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:12:24,206] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:12:24,207] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:12:24,232] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:12:24,314] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:12:24,340] - [task:165] INFO -  {'uptime': '687', 'memoryTotal': 15466930176, 'memoryFree': 12994297856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:12:24,365] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:12:24,392] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:12:24,392] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:12:24,444] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:12:24,448] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:24,448] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:25,090] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:25,091] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:12:25,215] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:12:25,216] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:12:25,248] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:12:25,249] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:12:25,276] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:12:25,361] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:12:25,388] - [task:165] INFO -  {'uptime': '687', 'memoryTotal': 15466930176, 'memoryFree': 12992966656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:12:25,414] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:12:25,444] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:12:25,445] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:12:25,495] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:12:25,498] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:25,499] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:26,126] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:26,127] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:12:26,251] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:12:26,252] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:12:26,281] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:12:26,282] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:12:26,307] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:12:26,364] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:12:26,859] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:12:31,864] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:12:31,953] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:12:31,960] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:31,960] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:32,586] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:32,587] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:12:32,706] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:12:32,707] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:12:32,707] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:12:33,508] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:12:33,568] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 00:12:33,569] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:12:34,621] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:12:34,960] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:12:35,253] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:12:35,257] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 00:12:35,314] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:35,315] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:35,940] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:35,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:35,945] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:36,783] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:36,792] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:36,792] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:37,885] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:37,896] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:37,896] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:38,951] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:45,224] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:12:45,225] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.63753172879125, 'mem_free': 12997611520, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:12:45,225] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:12:45,225] - [basetestcase:467] INFO - Time to execute basesetup : 23.98059606552124
[2022-09-12 00:12:45,280] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:12:45,280] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:12:45,337] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:12:45,337] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:12:45,391] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:12:45,392] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:12:45,449] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:12:45,449] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:12:45,505] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:12:45,566] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:12:45,566] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:12:45,566] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:12:50,577] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:12:50,580] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:12:50,580] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:12:51,174] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:12:52,213] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:12:52,504] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:12:54,628] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:12:54,713] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:12:54,713] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:12:54,714] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:13:24,743] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:13:24,771] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:13:24,796] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:13:24,863] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.245972ms
[2022-09-12 00:13:24,864] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '285e6486-d671-4ca1-a018-68824c1a8e1c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.245972ms', 'executionTime': '65.185221ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:13:24,864] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:13:24,891] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:13:24,917] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:13:25,644] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 723.985089ms
[2022-09-12 00:13:25,645] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:13:25,706] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:13:25,744] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:13:25,754] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.394366ms
[2022-09-12 00:13:25,977] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:13:26,016] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:13:26,032] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:13:26,033] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:13:26,054] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:13:26,056] - [base_gsi:326] INFO - []
[2022-09-12 00:13:26,938] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:13:26,966] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:13:27,019] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.432016ms
[2022-09-12 00:13:27,019] - [base_gsi:282] INFO - BUILD INDEX on default(employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr) USING GSI
[2022-09-12 00:13:28,050] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr) USING GSI
[2022-09-12 00:13:28,075] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr%29+USING+GSI
[2022-09-12 00:13:28,104] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 27.54642ms
[2022-09-12 00:13:29,136] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr'
[2022-09-12 00:13:29,161] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr%27
[2022-09-12 00:13:29,171] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.29441ms
[2022-09-12 00:13:30,201] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr'
[2022-09-12 00:13:30,226] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr%27
[2022-09-12 00:13:30,236] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.417225ms
[2022-09-12 00:13:31,267] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr'
[2022-09-12 00:13:31,293] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr%27
[2022-09-12 00:13:31,303] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.410127ms
[2022-09-12 00:13:31,331] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr ON default USING GSI
[2022-09-12 00:13:31,357] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr+ON+default+USING+GSI
[2022-09-12 00:13:31,404] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.003777ms
[2022-09-12 00:13:31,438] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr'
[2022-09-12 00:13:31,469] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee32eba3c9207741fdbd757a23ed90f194job_title_join_yr%27
[2022-09-12 00:13:31,476] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.31639ms
[2022-09-12 00:13:31,477] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ca04ca88-94cf-4f98-b040-829a2ebb4d44', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.31639ms', 'executionTime': '5.246989ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:13:31,580] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:13:31,583] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:31,584] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:32,230] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:32,287] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:13:32,287] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:13:32,390] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:13:32,449] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:13:32,453] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:32,453] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:33,066] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:33,121] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:13:33,121] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:13:33,227] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:13:33,282] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:13:33,283] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:13:33,283] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:13:33,309] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:13:33,334] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:13:33,343] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.340656ms
[2022-09-12 00:13:33,368] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:13:33,392] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:13:33,434] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.677685ms
[2022-09-12 00:13:33,496] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:13:33,497] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.22873636185319, 'mem_free': 12854464512, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:13:33,497] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:13:33,502] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:33,502] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:34,129] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:34,134] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:34,134] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:35,202] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:35,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:35,210] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:36,305] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:36,315] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:36,316] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:37,384] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:43,560] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-12 00:13:43,706] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:13:44,091] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:13:44,119] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:13:44,119] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:13:44,173] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:13:44,226] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:13:44,280] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:13:44,281] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:13:44,361] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:13:44,362] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:13:44,387] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:13:44,517] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:13:44,517] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:13:44,544] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:13:44,571] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:13:44,571] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:13:44,598] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:13:44,624] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:13:44,624] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:13:44,650] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:13:44,676] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:13:44,677] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:13:44,703] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:13:44,703] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 00:13:44,704] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:13:44,704] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 83.511s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_9'}
[2022-09-12 00:13:44,787] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:44,787] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:45,406] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:45,439] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:13:45,515] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:13:45,516] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-12 00:13:45,516] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:13:45,814] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:13:45,840] - [task:165] INFO -  {'uptime': '772', 'memoryTotal': 15466930176, 'memoryFree': 12990644224, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:13:45,865] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:13:45,865] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:13:45,866] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:13:45,894] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:13:45,928] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:13:45,929] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:13:45,955] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:13:45,956] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:13:45,956] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:13:45,956] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:13:46,005] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:13:46,008] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:46,008] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:46,655] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:46,656] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:13:46,768] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:13:46,769] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:13:46,798] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:13:46,799] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:13:46,826] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:13:46,910] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:13:46,936] - [task:165] INFO -  {'uptime': '772', 'memoryTotal': 15466930176, 'memoryFree': 12989468672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:13:46,961] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:13:46,989] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:13:46,989] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:13:47,039] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:13:47,042] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:47,042] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:47,633] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:47,634] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:13:47,748] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:13:47,749] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:13:47,780] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:13:47,781] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:13:47,809] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:13:47,892] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:13:47,919] - [task:165] INFO -  {'uptime': '768', 'memoryTotal': 15466930176, 'memoryFree': 12989693952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:13:47,944] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:13:47,972] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:13:47,972] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:13:48,021] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:13:48,026] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:48,027] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:48,607] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:48,608] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:13:48,724] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:13:48,725] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:13:48,754] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:13:48,755] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:13:48,782] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:13:48,871] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:13:48,898] - [task:165] INFO -  {'uptime': '767', 'memoryTotal': 15466930176, 'memoryFree': 12989587456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:13:48,924] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:13:48,951] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:13:48,951] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:13:49,001] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:13:49,004] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:49,004] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:49,618] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:49,619] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:13:49,735] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:13:49,737] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:13:49,767] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:13:49,768] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:13:49,793] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:13:49,850] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:13:50,493] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:13:55,498] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:13:55,582] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:13:55,588] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:55,588] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:56,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:56,242] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:13:56,362] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:13:56,364] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:13:56,364] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:13:56,970] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:13:57,025] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 00:13:57,025] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:13:57,967] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:13:58,276] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:13:58,564] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:13:58,568] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 00:13:58,630] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:58,631] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:13:59,273] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:13:59,279] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:13:59,279] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:14:00,198] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:14:00,210] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:14:00,211] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:14:01,345] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:14:01,355] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:14:01,355] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:14:02,499] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:14:08,896] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:14:08,896] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.74664127033418, 'mem_free': 13000671232, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:14:08,897] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:14:08,897] - [basetestcase:467] INFO - Time to execute basesetup : 24.11225938796997
[2022-09-12 00:14:08,951] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:14:08,951] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:14:09,004] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:14:09,005] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:14:09,057] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:14:09,057] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:14:09,109] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:14:09,109] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:14:09,160] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:09,222] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:14:09,222] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:14:09,222] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:14:14,234] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:14:14,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:14:14,238] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:14:14,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:14:15,981] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:14:16,148] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:14:18,302] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:14:18,375] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:18,376] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:14:18,376] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:14:48,381] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:14:48,411] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:14:48,436] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:14:48,502] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.122687ms
[2022-09-12 00:14:48,502] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a12a96dc-5a58-4a41-a17e-2f7f5a380ed5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.122687ms', 'executionTime': '64.051757ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:14:48,502] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:14:48,527] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:14:48,552] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:14:49,266] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 712.915215ms
[2022-09-12 00:14:49,267] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:14:49,324] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:14:49,357] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:14:49,365] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.40808ms
[2022-09-12 00:14:49,552] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:49,589] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:14:49,609] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:14:49,610] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:14:49,622] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:14:49,705] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:50,521] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:14:50,546] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:14:50,612] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.379445ms
[2022-09-12 00:14:50,612] - [base_gsi:282] INFO - BUILD INDEX on default(`employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr`) USING GSI
[2022-09-12 00:14:51,643] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr`) USING GSI
[2022-09-12 00:14:51,668] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr%60%29+USING+GSI
[2022-09-12 00:14:51,690] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.819714ms
[2022-09-12 00:14:51,736] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr'
[2022-09-12 00:14:51,765] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr%27
[2022-09-12 00:14:51,773] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.755775ms
[2022-09-12 00:14:52,805] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr'
[2022-09-12 00:14:52,831] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr%27
[2022-09-12 00:14:52,839] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.674161ms
[2022-09-12 00:14:53,869] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr'
[2022-09-12 00:14:53,899] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr%27
[2022-09-12 00:14:53,905] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.424268ms
[2022-09-12 00:14:54,936] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:14:54,962] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 00:14:54,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.893085ms
[2022-09-12 00:14:54,967] - [base_gsi:504] INFO - {'requestID': '38fb3134-30c3-47a5-ac43-42914eeb2150', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr', 'index_id': '3e7d7241a44c743c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.893085ms', 'executionTime': '2.813386ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 00:14:54,968] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:14:54,968] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:14:54,968] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:14:54,968] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:14:54,969] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:14:54,970] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:14:54,970] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:14:54,970] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:14:55,031] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:14:55,072] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:14:55,099] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 00:14:55,268] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 155.995369ms
[2022-09-12 00:14:55,269] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:14:55,269] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:14:57,609] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:57,634] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr'
[2022-09-12 00:14:57,659] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee99bacaffc1f9437faac6fa5b6cbda8cejoin_yr%27
[2022-09-12 00:14:57,664] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.238793ms
[2022-09-12 00:14:57,665] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1c408c70-3999-4da7-99f8-da1f238e0633', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.238793ms', 'executionTime': '4.178972ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:14:57,765] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:57,768] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:14:57,768] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:14:58,506] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:14:58,563] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:14:58,563] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:14:58,670] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:14:58,729] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:58,732] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:14:58,732] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:14:59,397] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:14:59,455] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:14:59,455] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:14:59,567] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:14:59,626] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:14:59,626] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:14:59,626] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:14:59,653] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:14:59,679] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:14:59,688] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.864016ms
[2022-09-12 00:14:59,688] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '89cf44a2-e9ca-48ef-bec0-979a3298071b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.864016ms', 'executionTime': '6.790865ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:14:59,747] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:14:59,747] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.25232181455722, 'mem_free': 12846239744, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:14:59,748] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:14:59,751] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:14:59,752] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:00,431] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:00,436] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:00,436] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:01,101] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:01,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:01,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:02,323] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:02,336] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:02,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:03,518] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:10,539] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-12 00:15:10,679] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:15:10,731] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:15:10,784] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:15:10,837] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:15:10,837] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:15:10,915] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:15:10,916] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:15:10,941] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:15:11,069] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:15:11,069] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:15:11,096] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:15:11,122] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:15:11,122] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:15:11,149] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:15:11,174] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:15:11,174] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:15:11,201] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:15:11,228] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:15:11,228] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:15:11,255] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:15:11,255] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 00:15:11,255] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:15:11,256] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.524s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_10'}
[2022-09-12 00:15:11,337] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:11,337] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:12,021] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:12,054] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:15:12,134] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:15:12,134] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-12 00:15:12,135] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:15:12,363] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:15:12,390] - [task:165] INFO -  {'uptime': '855', 'memoryTotal': 15466930176, 'memoryFree': 12991791104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:15:12,416] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:15:12,416] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:15:12,417] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:15:12,450] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:15:12,486] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:15:12,487] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:15:12,515] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:15:12,517] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:15:12,517] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:15:12,517] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:15:12,564] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:15:12,567] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:12,568] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:13,226] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:13,228] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:15:13,350] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:15:13,351] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:15:13,381] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:15:13,382] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:15:13,408] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:15:13,491] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:15:13,517] - [task:165] INFO -  {'uptime': '858', 'memoryTotal': 15466930176, 'memoryFree': 12991762432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:15:13,542] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:15:13,570] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:15:13,570] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:15:13,620] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:15:13,626] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:13,626] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:14,256] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:14,257] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:15:14,384] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:15:14,385] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:15:14,415] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:15:14,415] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:15:14,440] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:15:14,522] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:15:14,548] - [task:165] INFO -  {'uptime': '858', 'memoryTotal': 15466930176, 'memoryFree': 12991127552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:15:14,573] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:15:14,600] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:15:14,601] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:15:14,654] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:15:14,657] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:14,657] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:15,273] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:15,274] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:15:15,393] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:15:15,394] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:15:15,423] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:15:15,423] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:15:15,448] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:15:15,530] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:15:15,555] - [task:165] INFO -  {'uptime': '852', 'memoryTotal': 15466930176, 'memoryFree': 12991873024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:15:15,581] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:15:15,610] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:15:15,611] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:15:15,666] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:15:15,669] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:15,669] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:16,284] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:16,286] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:15:16,410] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:15:16,411] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:15:16,440] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:15:16,441] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:15:16,466] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:15:16,521] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:15:17,161] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:15:22,166] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:15:22,249] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:15:22,263] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:22,263] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:22,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:22,912] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:15:23,046] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:15:23,047] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:15:23,048] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:15:23,637] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:15:23,694] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 00:15:23,694] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:15:24,607] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:15:24,926] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:15:25,213] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:15:25,216] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 00:15:25,276] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:25,277] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:25,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:25,927] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:25,927] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:26,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:26,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:26,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:27,951] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:27,959] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:27,959] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:29,094] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:35,793] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:15:35,793] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.09346533983259, 'mem_free': 12987289600, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:15:35,794] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:15:35,794] - [basetestcase:467] INFO - Time to execute basesetup : 24.459922552108765
[2022-09-12 00:15:35,846] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:15:35,846] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:15:35,898] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:15:35,898] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:15:35,951] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:15:35,952] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:15:36,004] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:15:36,005] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:15:36,055] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:15:36,117] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:15:36,118] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:15:36,118] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:15:41,127] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:15:41,130] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:15:41,130] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:15:41,812] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:15:42,875] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:15:43,167] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:15:45,273] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:15:45,353] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:15:45,354] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:15:45,354] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:16:15,384] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:16:15,414] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:16:15,441] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:16:15,511] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.373571ms
[2022-09-12 00:16:15,511] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '341c596a-c8f0-4ce6-9872-4b2a22a4e05b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.373571ms', 'executionTime': '67.295261ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:16:15,512] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:16:15,543] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:16:15,571] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:16:16,256] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 683.333874ms
[2022-09-12 00:16:16,256] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:16:16,327] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:16:16,367] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:16:16,375] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.414519ms
[2022-09-12 00:16:16,579] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:16:16,619] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:16:16,636] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:16:16,636] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:16:16,656] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:16:16,733] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:16:17,538] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee62eb2c11a02543509641124c209d0717join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:16:17,563] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee62eb2c11a02543509641124c209d0717join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:16:17,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.89488ms
[2022-09-12 00:16:17,624] - [base_gsi:282] INFO - BUILD INDEX on default(`employee62eb2c11a02543509641124c209d0717join_yr`) USING GSI
[2022-09-12 00:16:18,654] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee62eb2c11a02543509641124c209d0717join_yr`) USING GSI
[2022-09-12 00:16:18,679] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee62eb2c11a02543509641124c209d0717join_yr%60%29+USING+GSI
[2022-09-12 00:16:18,710] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.660089ms
[2022-09-12 00:16:18,751] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee62eb2c11a02543509641124c209d0717join_yr'
[2022-09-12 00:16:18,782] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee62eb2c11a02543509641124c209d0717join_yr%27
[2022-09-12 00:16:18,791] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.698046ms
[2022-09-12 00:16:19,822] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee62eb2c11a02543509641124c209d0717join_yr'
[2022-09-12 00:16:19,847] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee62eb2c11a02543509641124c209d0717join_yr%27
[2022-09-12 00:16:19,857] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.205865ms
[2022-09-12 00:16:20,887] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee62eb2c11a02543509641124c209d0717join_yr'
[2022-09-12 00:16:20,913] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee62eb2c11a02543509641124c209d0717join_yr%27
[2022-09-12 00:16:20,923] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.869483ms
[2022-09-12 00:16:21,955] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:16:21,980] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 00:16:21,985] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.658334ms
[2022-09-12 00:16:21,985] - [base_gsi:504] INFO - {'requestID': '4972f3a9-f213-4678-8295-79c2a93ad1a1', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee62eb2c11a02543509641124c209d0717join_yr', 'index_id': '58dd25c2aa0eeb39', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.658334ms', 'executionTime': '2.597241ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 00:16:21,985] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:16:21,986] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:16:21,986] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:16:21,986] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:16:21,986] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:16:21,987] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:16:21,987] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:16:21,987] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:16:22,049] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:16:22,098] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:16:22,125] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 00:16:22,286] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 148.095216ms
[2022-09-12 00:16:22,286] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:16:22,287] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:16:23,488] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-12 00:16:23,541] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-12 00:16:23,571] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:16:23,598] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 00:16:23,602] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.401066ms
[2022-09-12 00:16:23,602] - [base_gsi:504] INFO - {'requestID': '61549cea-38ab-486b-9094-a59cd9b1aef7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee62eb2c11a02543509641124c209d0717join_yr', 'index_id': '58dd25c2aa0eeb39', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.401066ms', 'executionTime': '2.334775ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 00:16:23,602] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:16:23,603] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:16:23,603] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:16:23,603] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:16:23,603] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:16:23,603] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:16:23,603] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:16:23,604] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:16:23,657] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:16:23,697] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:16:23,721] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 00:16:23,840] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 109.772951ms
[2022-09-12 00:16:23,840] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:16:23,841] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:16:25,046] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee62eb2c11a02543509641124c209d0717join_yr ON default USING GSI
[2022-09-12 00:16:25,074] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee62eb2c11a02543509641124c209d0717join_yr+ON+default+USING+GSI
[2022-09-12 00:16:25,122] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.247568ms
[2022-09-12 00:16:25,155] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee62eb2c11a02543509641124c209d0717join_yr'
[2022-09-12 00:16:25,186] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee62eb2c11a02543509641124c209d0717join_yr%27
[2022-09-12 00:16:25,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.632497ms
[2022-09-12 00:16:25,193] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b4e42c96-84ac-411a-a380-b9feb8b19ac5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.632497ms', 'executionTime': '5.561338ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:16:25,298] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:16:25,301] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:25,301] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:26,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:26,062] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:16:26,062] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:16:26,172] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:16:26,231] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:16:26,234] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:26,235] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:26,903] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:26,957] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:16:26,958] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:16:27,085] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:16:27,141] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:16:27,141] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:16:27,141] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:16:27,167] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:16:27,192] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:16:27,205] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.971719ms
[2022-09-12 00:16:27,236] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:16:27,261] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:16:27,320] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.187975ms
[2022-09-12 00:16:27,385] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:16:27,385] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.22958305397104, 'mem_free': 12828442624, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:16:27,386] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:16:27,390] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:27,391] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:28,083] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:28,088] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:28,088] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:28,775] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:28,779] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:28,780] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:29,861] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:29,873] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:29,874] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:31,069] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:38,572] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-12 00:16:38,711] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:16:39,185] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:16:39,213] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:16:39,214] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:16:39,268] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:16:39,324] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:16:39,377] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:16:39,378] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:16:39,462] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:16:39,463] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:16:39,490] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:16:39,628] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:16:39,628] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:16:39,656] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:16:39,681] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:16:39,681] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:16:39,710] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:16:39,737] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:16:39,737] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:16:39,764] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:16:39,791] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:16:39,791] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:16:39,821] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:16:39,821] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 00:16:39,822] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:16:39,823] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 88.558s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_11'}
[2022-09-12 00:16:39,927] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:39,928] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:40,703] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:40,737] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:16:40,817] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:16:40,818] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-12 00:16:40,819] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:16:40,951] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:16:40,979] - [task:165] INFO -  {'uptime': '947', 'memoryTotal': 15466930176, 'memoryFree': 12975337472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:16:41,007] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:16:41,007] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:16:41,007] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:16:41,040] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:16:41,084] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:16:41,085] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:16:41,114] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:16:41,115] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:16:41,115] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:16:41,115] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:16:41,163] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:16:41,166] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:41,166] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:41,818] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:41,819] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:16:41,940] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:16:41,941] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:16:41,971] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:16:41,972] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:16:41,998] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:16:42,084] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:16:42,109] - [task:165] INFO -  {'uptime': '943', 'memoryTotal': 15466930176, 'memoryFree': 12974628864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:16:42,134] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:16:42,162] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:16:42,163] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:16:42,213] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:16:42,216] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:42,216] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:42,883] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:42,884] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:16:43,014] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:16:43,016] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:16:43,046] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:16:43,046] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:16:43,072] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:16:43,156] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:16:43,183] - [task:165] INFO -  {'uptime': '943', 'memoryTotal': 15466930176, 'memoryFree': 12974628864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:16:43,208] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:16:43,236] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:16:43,236] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:16:43,287] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:16:43,291] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:43,291] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:43,992] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:43,994] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:16:44,122] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:16:44,123] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:16:44,153] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:16:44,153] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:16:44,179] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:16:44,266] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:16:44,295] - [task:165] INFO -  {'uptime': '943', 'memoryTotal': 15466930176, 'memoryFree': 12975366144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:16:44,320] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:16:44,351] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:16:44,351] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:16:44,403] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:16:44,407] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:44,407] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:45,049] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:45,051] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:16:45,174] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:16:45,175] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:16:45,205] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:16:45,206] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:16:45,231] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:16:45,286] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:16:45,932] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:16:50,938] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:16:51,021] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:16:51,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:51,031] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:51,688] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:51,689] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:16:51,818] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:16:51,819] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:16:51,819] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:16:52,428] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:16:52,486] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 00:16:52,487] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:16:53,413] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:16:53,714] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:16:53,972] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:16:53,975] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 00:16:54,030] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:54,030] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:54,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:54,688] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:54,688] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:55,626] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:55,634] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:55,634] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:56,850] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:16:56,858] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:16:56,858] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:16:58,045] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:17:05,194] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:17:05,194] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.45945113037273, 'mem_free': 12971425792, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:17:05,194] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:17:05,195] - [basetestcase:467] INFO - Time to execute basesetup : 25.271767139434814
[2022-09-12 00:17:05,245] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:17:05,245] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:17:05,299] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:17:05,299] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:17:05,351] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:17:05,352] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:17:05,404] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:17:05,404] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:17:05,455] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:17:05,511] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:17:05,511] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:17:05,512] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:17:10,524] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:17:10,527] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:17:10,528] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:17:11,174] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:17:12,344] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:17:12,515] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:17:15,617] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:17:15,691] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:17:15,692] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:17:15,692] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:17:45,702] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:17:45,731] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:17:45,757] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:17:45,822] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.020768ms
[2022-09-12 00:17:45,823] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3b2c94f9-7ed9-4e5b-ab00-f8b8c5bc6d02', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.020768ms', 'executionTime': '63.944474ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:17:45,823] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:17:45,848] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:17:45,873] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:17:46,624] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 749.407518ms
[2022-09-12 00:17:46,625] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:17:46,688] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:17:46,725] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:17:46,733] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.567591ms
[2022-09-12 00:17:46,944] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:17:46,980] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:17:46,995] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:17:46,997] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:17:47,015] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:17:47,087] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:17:47,902] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 00:17:47,927] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 00:17:47,980] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.503622ms
[2022-09-12 00:17:47,980] - [base_gsi:282] INFO - BUILD INDEX on default(`employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr`) USING GSI
[2022-09-12 00:17:49,011] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr`) USING GSI
[2022-09-12 00:17:49,036] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr%60%29+USING+GSI
[2022-09-12 00:17:49,057] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.040013ms
[2022-09-12 00:17:49,099] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr'
[2022-09-12 00:17:49,128] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr%27
[2022-09-12 00:17:49,137] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.138377ms
[2022-09-12 00:17:50,167] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr'
[2022-09-12 00:17:50,193] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr%27
[2022-09-12 00:17:50,204] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.237787ms
[2022-09-12 00:17:51,236] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr'
[2022-09-12 00:17:51,261] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr%27
[2022-09-12 00:17:51,271] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.153958ms
[2022-09-12 00:17:52,304] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:17:52,330] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 00:17:52,336] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.870363ms
[2022-09-12 00:17:52,336] - [base_gsi:504] INFO - {'requestID': 'd8be5043-d000-41fb-b047-e2076246886b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr', 'index_id': 'cd26926ebac6bd5d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.870363ms', 'executionTime': '3.806264ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 00:17:52,337] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 00:17:52,337] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:17:52,337] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 00:17:52,338] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 00:17:52,338] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:17:52,338] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:17:52,338] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:17:52,339] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 00:17:52,399] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 00:17:52,439] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 00:17:52,465] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 00:17:52,626] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 147.541932ms
[2022-09-12 00:17:52,627] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 00:17:52,627] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 00:17:54,945] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 00:17:57,362] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:17:57,416] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 00:17:57,416] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:17:58,324] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:17:58,735] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:17:59,016] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:17:59,021] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 00:18:01,053] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:18:01,053] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:01,080] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:18:01,109] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:18:01,109] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:01,135] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:18:01,163] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:18:01,164] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:01,190] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:18:01,218] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:18:01,218] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:01,246] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:18:01,296] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:18:01,321] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr'
[2022-09-12 00:18:01,347] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr%27
[2022-09-12 00:18:01,395] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.443832ms
[2022-09-12 00:18:01,395] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '368ce0bb-64e4-46a7-8fbf-ca20c11bce48', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '46.443832ms', 'executionTime': '46.370993ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:18:01,422] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr'
[2022-09-12 00:18:01,446] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefed4a475d95647de93bfc6b253a8dc5ajoin_yr%27
[2022-09-12 00:18:01,449] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.156213ms
[2022-09-12 00:18:01,449] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'cb0a5069-08fc-4375-b1ab-1292f2d89c45', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.156213ms', 'executionTime': '1.096334ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:18:01,500] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:18:01,630] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:18:01,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:01,632] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:02,350] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:02,409] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:18:02,409] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:18:02,525] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:18:02,580] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:18:02,583] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:02,584] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:03,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:03,326] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:18:03,326] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:18:03,440] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:18:03,499] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:18:03,500] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:18:03,500] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:18:03,525] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:18:03,550] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:18:03,559] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.724424ms
[2022-09-12 00:18:03,559] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e7a2f4d9-0fac-4da8-8734-d01cf8eda95e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.724424ms', 'executionTime': '7.612748ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:18:03,560] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:18:03,585] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:18:03,611] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:18:03,613] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.174899ms
[2022-09-12 00:18:03,613] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9b61f74e-9de6-490d-8632-c86b29473107', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.174899ms', 'executionTime': '1.103177ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:18:03,665] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:18:03,665] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.65964679721414, 'mem_free': 12804321280, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:18:03,665] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:18:03,668] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:03,669] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:04,348] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:04,353] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:04,353] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:05,030] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:05,035] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:05,035] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:06,255] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:06,263] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:06,264] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:07,508] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:15,165] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 00:18:15,305] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:18:15,608] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:18:15,636] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:18:15,636] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:18:15,692] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:18:15,746] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:18:15,799] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:18:15,799] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:18:15,878] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:18:15,879] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:18:15,905] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:16,032] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:18:16,033] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:16,059] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:18:16,085] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:18:16,085] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:16,111] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:18:16,137] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:18:16,137] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:16,163] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:18:16,189] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:18:16,189] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:16,216] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:18:16,216] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 00:18:16,217] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:18:16,217] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.347s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_00-00-49/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-12 00:18:16,276] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:16,276] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:16,960] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:16,994] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:18:17,073] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 00:18:17,073] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-12 00:18:17,074] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 00:18:17,302] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 00:18:17,328] - [task:165] INFO -  {'uptime': '1043', 'memoryTotal': 15466930176, 'memoryFree': 12952203264, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:18:17,354] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 00:18:17,354] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 00:18:17,355] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 00:18:17,395] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 00:18:17,428] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 00:18:17,429] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 00:18:17,457] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 00:18:17,458] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 00:18:17,458] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 00:18:17,459] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 00:18:17,506] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:18:17,509] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:17,509] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:18,188] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:18,189] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:18:18,314] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:18:18,315] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:18:18,345] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:18:18,346] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:18:18,371] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:18:18,454] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 00:18:18,482] - [task:165] INFO -  {'uptime': '1043', 'memoryTotal': 15466930176, 'memoryFree': 12949835776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:18:18,508] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:18:18,536] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 00:18:18,536] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 00:18:18,586] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:18:18,589] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:18,589] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:19,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:19,271] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:18:19,405] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:18:19,407] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:18:19,442] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:18:19,442] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:18:19,470] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:18:19,556] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 00:18:19,584] - [task:165] INFO -  {'uptime': '1039', 'memoryTotal': 15466930176, 'memoryFree': 12950142976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:18:19,611] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:18:19,639] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 00:18:19,640] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 00:18:19,690] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:18:19,693] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:19,693] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:20,377] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:20,379] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:18:20,507] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:18:20,508] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:18:20,537] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:18:20,538] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:18:20,562] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:18:20,649] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 00:18:20,675] - [task:165] INFO -  {'uptime': '1038', 'memoryTotal': 15466930176, 'memoryFree': 12949913600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 00:18:20,700] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 00:18:20,728] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 00:18:20,729] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 00:18:20,778] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 00:18:20,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:20,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:21,508] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:21,509] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:18:21,647] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:18:21,648] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:18:21,678] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 00:18:21,679] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 00:18:21,705] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 00:18:21,764] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 00:18:22,289] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:18:27,294] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 00:18:27,379] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 00:18:27,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:27,384] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:28,078] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:28,079] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 00:18:28,205] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 00:18:28,206] - [remote_util:5249] INFO - b'ok'
[2022-09-12 00:18:28,207] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 00:18:28,887] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 00:18:28,941] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 00:18:28,941] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 00:18:29,822] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:18:30,136] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:18:30,393] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 00:18:30,396] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-12 00:18:30,450] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:30,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:31,115] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:31,119] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:31,119] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:32,215] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:32,223] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:32,224] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:33,481] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:33,489] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:33,489] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:34,685] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:41,984] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:18:41,984] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.68066827169358, 'mem_free': 12950581248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:18:41,984] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:18:41,985] - [basetestcase:467] INFO - Time to execute basesetup : 25.711769104003906
[2022-09-12 00:18:42,035] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:18:42,035] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:18:42,088] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:18:42,088] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:18:42,142] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:18:42,142] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:18:42,195] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 00:18:42,195] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 00:18:42,246] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:18:42,310] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 00:18:42,311] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 00:18:42,311] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 00:18:47,318] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 00:18:47,321] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:18:47,321] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:18:48,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:18:49,191] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 00:18:49,364] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 00:18:51,517] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 00:18:51,593] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:18:51,594] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 00:18:51,594] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 00:19:21,604] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 00:19:21,634] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:19:21,659] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:19:21,730] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.866873ms
[2022-09-12 00:19:21,730] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2c7ab635-66dc-4837-8827-1ebfe763ce52', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.866873ms', 'executionTime': '68.807651ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 00:19:21,730] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 00:19:21,758] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 00:19:21,782] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 00:19:22,466] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 682.142672ms
[2022-09-12 00:19:22,467] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 00:19:22,552] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:19:22,589] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:19:22,598] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.960968ms
[2022-09-12 00:19:22,790] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:19:22,827] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 00:19:22,845] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 00:19:22,845] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 00:19:22,864] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 00:19:22,937] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:19:23,075] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:19:23,078] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:19:23,078] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:19:23,875] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:19:23,933] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:19:23,934] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 00:19:24,059] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:19:24,121] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:19:24,125] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:19:24,125] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:19:24,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:19:24,979] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 00:19:24,979] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 00:19:25,108] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 00:19:25,167] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 00:19:25,167] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 00:19:25,168] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:19:25,193] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 00:19:25,218] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 00:19:25,228] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.590503ms
[2022-09-12 00:19:25,254] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 00:19:25,282] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 00:19:25,338] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.622501ms
[2022-09-12 00:19:25,406] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 00:19:25,407] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.538499725608175, 'mem_free': 12778704896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 00:19:25,407] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 00:19:25,411] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:19:25,411] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:19:26,127] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:19:26,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:19:26,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:19:27,384] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:19:27,392] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:19:27,392] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:19:28,689] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:19:28,702] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 00:19:28,702] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 00:19:29,991] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 00:19:37,410] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-12 00:19:37,556] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 00:19:38,472] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 00:19:38,500] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 00:19:38,500] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 00:19:38,554] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:19:38,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:19:38,660] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 00:19:38,661] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 00:19:38,754] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 00:19:38,755] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 00:19:38,780] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:19:38,911] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 00:19:38,911] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:19:38,938] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 00:19:38,964] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 00:19:38,964] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:19:38,991] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 00:19:39,017] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 00:19:39,017] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:19:39,043] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 00:19:39,068] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 00:19:39,068] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 00:19:39,095] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 00:19:39,095] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-12 00:19:39,095] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 00:19:39,096] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 82.902s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 12206 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-11.09.2022-22.30.cfg
Build Log: make-11.09.2022-22.30.log
Server Log: logs-11.09.2022-22.30.tar.gz

Finished