Building

Started building at 2022/09/03 01:31:01
Using pegged server, 1739 build
Calculating base
Updating mirror
Basing run on 7.2.0-1739 eddcdf9fd
Updating tree for run 03.09.2022-01.31
query is at b2ca880, changes since last good build: none
gometa is at a237ed7, changes since last good build: 

ns_server is at f2c62fe, changes since last good build: none
couchstore is at 20545dc, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at fde39ad, changes since last good build: none
Switching indexing to unstable
indexing is at 9f5873f, changes since last good build: none
Switching plasma to unstable
plasma is at cfa6534, changes since last good build: none
Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at e1ca021, changes since last good build: 
 e1ca0215b Test cases for compute unit and sequential scan
Pulling in uncommitted change 176523 at refs/changes/23/176523/3
Total 51 (delta 40), reused 48 (delta 40)
[unstable 99ecae3b] MB-51825: Fix memedb_slice_test adding numVBuckets arg
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Jun 22 16:31:49 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178917 at refs/changes/17/178917/8
Total 9 (delta 6), reused 7 (delta 6)
[unstable faf1ffea] MB-51825: Pass numVBuckets to Storage from indexer
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Aug 17 15:08:14 2022 +0530
 5 files changed, 56 insertions(+), 26 deletions(-)
Pulling in uncommitted change 178948 at refs/changes/48/178948/10
Total 15 (delta 10), reused 12 (delta 10)
[unstable 4186544b] MB-51825: Indexer - Fetch numVBuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Aug 17 20:53:00 2022 +0530
 2 files changed, 57 insertions(+), 21 deletions(-)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 011084f] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/03 02:18:36
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Sep 03 02:23:59 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 03 02:27:53 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 03 02:28:38 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 03 02:29:56 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 03 02:30:47 b" 'stop_on_failure': 'True'}"
Sep 03 02:30:47 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 03 02:30:47 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_1'}"
Sep 03 02:30:47 b'-->result: '
Sep 03 02:30:47 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 03 02:30:47 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 03 02:30:47 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_2'}"
Sep 03 02:30:47 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 03 02:30:47 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 03 02:30:47 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_3'}"
Sep 03 02:30:47 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 03 02:30:47 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 03 02:30:47 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 03 02:30:47 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_4'}"
Sep 03 02:30:47 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 03 02:30:47 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 03 02:30:47 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 03 02:40:27 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 03 02:41:12 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 03 02:44:55 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 03 02:48:50 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 03 02:51:32 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 03 02:51:32 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_5'}"
Sep 03 02:51:32 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 03 02:51:32 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 03 02:51:32 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 03 02:51:32 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_6'}"
Sep 03 02:51:32 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 03 02:51:32 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 03 02:51:32 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 03 02:51:32 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_7'}"
Sep 03 02:51:32 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 03 02:51:32 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 03 02:51:32 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 03 02:51:32 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 03 02:51:32 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_02-18-58/test_8'}"
Sep 03 02:51:32 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 03 02:51:32 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 03 02:51:32 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 03 02:51:32 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 03 02:51:32 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 03 02:51:33 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 03 02:51:33 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 03 02:51:33 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 03 02:51:33 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 03 02:51:33 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 03 02:51:33 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 03 02:51:33 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 03 02:51:33 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

=== RUN   TestMerger
--- PASS: TestMerger (0.02s)
=== RUN   TestInsert
--- PASS: TestInsert (0.00s)
=== RUN   TestInsertPerf
16000000 items took 30.292625626s -> 528181.35336104 items/s conflicts 24
--- PASS: TestInsertPerf (30.29s)
=== RUN   TestGetPerf
16000000 items took 5.37995384s -> 2.974003211893729e+06 items/s
--- PASS: TestGetPerf (5.98s)
=== RUN   TestGetRangeSplitItems
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3351,
"memory_used":            37360992,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749144,
"level1": 187776,
"level2": 47222,
"level3": 11903,
"level4": 2974,
"level5": 735,
"level6": 182,
"level7": 48,
"level8": 12,
"level9": 2,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Split range keys [35911 108875 357987 462344 598654 671258 781160]
No of items in each range [35911 72964 249112 104357 136310 72604 109902 218840]
--- PASS: TestGetRangeSplitItems (0.60s)
=== RUN   TestBuilder
{
"node_count":             50000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3333,
"memory_used":            1866650896,
"node_allocs":            50000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 37500395,
"level1": 9374070,
"level2": 2344404,
"level3": 586129,
"level4": 146617,
"level5": 36338,
"level6": 9034,
"level7": 2298,
"level8": 539,
"level9": 123,
"level10": 39,
"level11": 9,
"level12": 5,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 7.742642321s to build 50000000 items, 6.4577435e+06 items/sec
Took 398.608844ms to iterate 50000000 items
--- PASS: TestBuilder (8.14s)
=== RUN   TestNodeDCAS
--- PASS: TestNodeDCAS (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb/skiplist	45.278s
=== RUN   TestInsert
Set IO Concurrency: 716
--- PASS: TestInsert (0.01s)
=== RUN   TestInsertPerf
20000000 items took 49.570154033s -> 403468.58689778403 items/s snapshots_created 210 live_snapshots 1
--- PASS: TestInsertPerf (61.66s)
=== RUN   TestInsertDuplicates
--- PASS: TestInsertDuplicates (0.01s)
=== RUN   TestGetPerf
16000000 items took 14.325135331s -> 1.1169178950355565e+06 items/s
--- PASS: TestGetPerf (15.87s)
=== RUN   TestLoadStoreDisk
Inserting 1000000 items took 1.477449729s
{
"node_count":             1001052,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       11,
"next_pointers_per_node": 1.3333,
"memory_used":            57338400,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750600,
"level1": 188016,
"level2": 46826,
"level3": 11767,
"level4": 2863,
"level5": 743,
"level6": 167,
"level7": 54,
"level8": 11,
"level9": 4,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
2022-09-03T02:54:03.109+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[42070 76357 40412 29464 73014 56240 80308 21449 28094 45805 56546 39966 24521 29283 34288 322183]] total count [1000000] snap count [1000000] size per shard [[336560 610856 323296 235712 584112 449920 642464 171592 224752 366440 452368 319728 196168 234264 274304 2577464]] total size [8000000] memoryInUse [57338400]
Storing to disk took 318.779336ms
Loading from disk took 301.809238ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3324,
"memory_used":            57318992,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750051,
"level1": 187716,
"level2": 46883,
"level3": 11653,
"level4": 2782,
"level5": 691,
"level6": 176,
"level7": 34,
"level8": 9,
"level9": 3,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestLoadStoreDisk (3.96s)
=== RUN   TestConcurrentLoadCloseFragmentation
Done Inserting 2000000 items
frag = NaN%
2022-09-03T02:54:08.902+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[2188 6470 23054 223965 100899 42235 52588 199529 90479 70379 121365 32412 281721 88119 50135 614462]] total count [2000000] snap count [2000000] size per shard [[17504 51760 184432 1791720 807192 337880 420704 1596232 723832 563032 970920 259296 2253768 704952 401080 4915696]] total size [16000000] memoryInUse [114638256]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentation (5.96s)
=== RUN   TestConcurrentLoadCloseFragmentationSmall
Done Inserting 10 items
frag = NaN%
2022-09-03T02:54:10.785+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationSmall (0.09s)
=== RUN   TestConcurrentLoadCloseFragmentationEmpty
Done Inserting 0 items
frag = NaN%
2022-09-03T02:54:10.877+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationEmpty (0.06s)
=== RUN   TestConcurrentCloseSingleNode
A
--- PASS: TestConcurrentCloseSingleNode (0.11s)
=== RUN   TestStoreDiskShutdown
Inserting 1000000 items took 1.751895029s
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       18,
"next_pointers_per_node": 1.3335,
"memory_used":            57335708,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749880,
"level1": 187777,
"level2": 46545,
"level3": 11883,
"level4": 2943,
"level5": 733,
"level6": 168,
"level7": 50,
"level8": 15,
"level9": 5,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
Prepare returns error:  MemDB instance has been shutdown
--- PASS: TestStoreDiskShutdown (2.45s)
=== RUN   TestDelete
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.2000,
"memory_used":            572,
"node_allocs":            10,
"node_frees":             0,
"level_node_distribution":{
"level0": 8,
"level1": 2,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.0000,
"memory_used":            540,
"node_allocs":            20,
"node_frees":             10,
"level_node_distribution":{
"level0": 10,
"level1": 0,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestDelete (1.00s)
=== RUN   TestGCPerf
final_node_count = 16000, average_live_node_count = 68608, wait_time_for_collection = 21ms
--- PASS: TestGCPerf (36.49s)
=== RUN   TestMemoryInUse
ItemsCount: 5000, MemoryInUse: 296992, NodesCount: 5000
ItemsCount: 0, MemoryInUse: 64, NodesCount: 0
--- PASS: TestMemoryInUse (1.01s)
=== RUN   TestFullScan
Inserting 1600000 items took 2.580926816s
{
"node_count":             1600035,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       7,
"next_pointers_per_node": 1.3341,
"memory_used":            91756140,
"node_allocs":            1600000,
"node_frees":             0,
"level_node_distribution":{
"level0": 1199572,
"level1": 300095,
"level2": 75175,
"level3": 18793,
"level4": 4793,
"level5": 1192,
"level6": 313,
"level7": 83,
"level8": 16,
"level9": 2,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Full iteration of 1600000 items took 257.26237ms
--- PASS: TestFullScan (3.99s)
=== RUN   TestVisitor
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3334,
"memory_used":            57333696,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750109,
"level1": 187672,
"level2": 46484,
"level3": 11688,
"level4": 2946,
"level5": 837,
"level6": 189,
"level7": 54,
"level8": 18,
"level9": 3,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 55.387158ms to iterate 1000000 items, 1.8054726e+07 items/s
shard - 0 count = 8269, range: 0-8268
shard - 1 count = 64252, range: 8269-72520
shard - 2 count = 13851, range: 72521-86371
shard - 3 count = 40138, range: 86372-126509
shard - 4 count = 47455, range: 126510-173964
shard - 5 count = 5869, range: 173965-179833
shard - 6 count = 8735, range: 179834-188568
shard - 7 count = 17731, range: 188569-206299
shard - 8 count = 45522, range: 206300-251821
shard - 9 count = 3849, range: 251822-255670
shard - 10 count = 5753, range: 255671-261423
shard - 11 count = 39204, range: 261424-300627
shard - 12 count = 42993, range: 300628-343620
shard - 13 count = 26814, range: 343621-370434
shard - 14 count = 27393, range: 370435-397827
shard - 15 count = 52705, range: 397828-450532
shard - 16 count = 16902, range: 450533-467434
shard - 17 count = 9399, range: 467435-476833
shard - 18 count = 13835, range: 476834-490668
shard - 19 count = 7301, range: 490669-497969
shard - 20 count = 60375, range: 497970-558344
shard - 21 count = 20622, range: 558345-578966
shard - 22 count = 32427, range: 578967-611393
shard - 23 count = 15570, range: 611394-626963
shard - 24 count = 19007, range: 626964-645970
shard - 25 count = 44506, range: 645971-690476
shard - 26 count = 23483, range: 690477-713959
shard - 27 count = 72253, range: 713960-786212
shard - 28 count = 13627, range: 786213-799839
shard - 29 count = 4972, range: 799840-804811
shard - 30 count = 18731, range: 804812-823542
shard - 31 count = 176457, range: 823543-999999
--- PASS: TestVisitor (1.78s)
=== RUN   TestVisitorError
--- PASS: TestVisitorError (0.44s)
=== RUN   TestLoadDeltaStoreDisk
snap count = 1000000
2022-09-03T02:55:02.182+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 60921 245368 62374 106720 56894 57021 160974 201483]] total count [951755] snap count [1000000] size per shard [[0 0 0 0 0 0 0 0 487368 1962944 498992 853760 455152 456168 1287792 1611864]] total size [7614040] memoryInUse [127274776]
Storing to disk took 15.418974597s
Loading from disk took 806.358773ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3335,
"memory_used":            57335312,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749617,
"level1": 188118,
"level2": 46700,
"level3": 11624,
"level4": 2963,
"level5": 724,
"level6": 195,
"level7": 47,
"level8": 12,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Restored 48245
RestoredFailed 381257
--- PASS: TestLoadDeltaStoreDisk (18.19s)
=== RUN   TestExecuteConcurrGCWorkers
--- PASS: TestExecuteConcurrGCWorkers (0.57s)
=== RUN   TestCloseWithActiveIterators
--- PASS: TestCloseWithActiveIterators (14.94s)
=== RUN   TestDiskCorruption
Inserting 100000 items took 103.597184ms
{
"node_count":             100011,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       5,
"next_pointers_per_node": 1.3303,
"memory_used":            5728592,
"node_allocs":            100000,
"node_frees":             0,
"level_node_distribution":{
"level0": 75119,
"level1": 18784,
"level2": 4611,
"level3": 1101,
"level4": 283,
"level5": 86,
"level6": 23,
"level7": 4,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
2022-09-03T02:55:32.032+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[4899 6693 4884 7799 11043 12632 2838 8485 4605 2260 7268 6733 14262 2938 2661 0]] total count [100000] snap count [100000] size per shard [[39192 53544 39072 62392 88344 101056 22704 67880 36840 18080 58144 53864 114096 23504 21288 0]] total size [800000] memoryInUse [5728592]
Storing to disk took 92.811415ms
Loading from disk took 40.751186ms
--- PASS: TestDiskCorruption (0.30s)
=== RUN   TestSnapshotStats
--- PASS: TestSnapshotStats (0.00s)
=== RUN   TestInsertDeleteConcurrent
total items in snapshot: 7960158  items deleted: 2039842
{
"node_count":             7960158,
"soft_deletes":           0,
"read_conflicts":         4243,
"insert_conflicts":       108,
"next_pointers_per_node": 1.3334,
"memory_used":            456392216,
"node_allocs":            10000000,
"node_frees":             10000002,
"level_node_distribution":{
"level0": 5970437,
"level1": 1491595,
"level2": 373662,
"level3": 93275,
"level4": 23291,
"level5": 5945,
"level6": 1464,
"level7": 364,
"level8": 99,
"level9": 18,
"level10": 7,
"level11": 1,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
allocs: 20000002 frees: 20000002
--- PASS: TestInsertDeleteConcurrent (19.60s)
=== RUN   TestNodeList
--- PASS: TestNodeList (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb	188.779s
=== RUN   TestInteger
--- PASS: TestInteger (0.00s)
=== RUN   TestSmallDecimal
--- PASS: TestSmallDecimal (0.00s)
=== RUN   TestLargeDecimal
--- PASS: TestLargeDecimal (0.00s)
=== RUN   TestFloat
--- PASS: TestFloat (0.00s)
=== RUN   TestSuffixCoding
--- PASS: TestSuffixCoding (0.00s)
=== RUN   TestCodecLength
--- PASS: TestCodecLength (0.00s)
=== RUN   TestSpecialString
--- PASS: TestSpecialString (0.00s)
=== RUN   TestCodecNoLength
--- PASS: TestCodecNoLength (0.00s)
=== RUN   TestCodecJSON
--- PASS: TestCodecJSON (0.00s)
=== RUN   TestReference
--- PASS: TestReference (0.00s)
=== RUN   TestN1QLEncode
--- PASS: TestN1QLEncode (0.00s)
=== RUN   TestArrayExplodeJoin
--- PASS: TestArrayExplodeJoin (0.00s)
=== RUN   TestN1QLDecode
--- PASS: TestN1QLDecode (0.00s)
=== RUN   TestN1QLDecode2
--- PASS: TestN1QLDecode2 (0.00s)
=== RUN   TestArrayExplodeJoin2
--- PASS: TestArrayExplodeJoin2 (0.00s)
=== RUN   TestMB28956
--- PASS: TestMB28956 (0.00s)
=== RUN   TestFixEncodedInt
--- PASS: TestFixEncodedInt (0.00s)
=== RUN   TestN1QLDecodeLargeInt64
--- PASS: TestN1QLDecodeLargeInt64 (0.00s)
=== RUN   TestMixedModeFixEncodedInt
TESTING [4111686018427387900, -8223372036854775808, 822337203685477618] 
PASS 
TESTING [0] 
PASS 
TESTING [0.0] 
PASS 
TESTING [0.0000] 
PASS 
TESTING [0.0000000] 
PASS 
TESTING [-0] 
PASS 
TESTING [-0.0] 
PASS 
TESTING [-0.0000] 
PASS 
TESTING [-0.0000000] 
PASS 
TESTING [1] 
PASS 
TESTING [20] 
PASS 
TESTING [3456] 
PASS 
TESTING [7645000] 
PASS 
TESTING [9223372036854775807] 
PASS 
TESTING [9223372036854775806] 
PASS 
TESTING [9223372036854775808] 
PASS 
TESTING [92233720368547758071234000] 
PASS 
TESTING [92233720368547758071234987437653] 
PASS 
TESTING [12300000000000000000000000000000056] 
PASS 
TESTING [12300000000000000000000000000000000] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [12300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [210690] 
PASS 
TESTING [90000] 
PASS 
TESTING [123000000] 
PASS 
TESTING [3.60e2] 
PASS 
TESTING [36e2] 
PASS 
TESTING [1.9999999999e10] 
PASS 
TESTING [1.99999e10] 
PASS 
TESTING [1.99999e5] 
PASS 
TESTING [0.00000000000012e15] 
PASS 
TESTING [7.64507352e8] 
PASS 
TESTING [9.2233720368547758071234987437653e31] 
PASS 
TESTING [2650e-1] 
PASS 
TESTING [26500e-1] 
PASS 
TESTING [-1] 
PASS 
TESTING [-20] 
PASS 
TESTING [-3456] 
PASS 
TESTING [-7645000] 
PASS 
TESTING [-9223372036854775808] 
PASS 
TESTING [-9223372036854775807] 
PASS 
TESTING [-9223372036854775806] 
PASS 
TESTING [-9223372036854775809] 
PASS 
TESTING [-92233720368547758071234000] 
PASS 
TESTING [-92233720368547758071234987437653] 
PASS 
TESTING [-12300000000000000000000000000000056] 
PASS 
TESTING [-12300000000000000000000000000000000] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [-210690] 
PASS 
TESTING [-90000] 
PASS 
TESTING [-123000000] 
PASS 
TESTING [-3.60e2] 
PASS 
TESTING [-36e2] 
PASS 
TESTING [-1.9999999999e10] 
PASS 
TESTING [-1.99999e10] 
PASS 
TESTING [-1.99999e5] 
PASS 
TESTING [-0.00000000000012e15] 
PASS 
TESTING [-2650e-1] 
PASS 
TESTING [-26500e-1] 
PASS 
TESTING [0.03] 
PASS 
TESTING [198.60] 
PASS 
TESTING [2000045.178] 
PASS 
TESTING [1.7976931348623157e+308] 
PASS 
TESTING [0.000000000000000000890] 
PASS 
TESTING [257953786.9864236576] 
PASS 
TESTING [257953786.9864236576e8] 
PASS 
TESTING [36.912e3] 
PASS 
TESTING [2761.67e0] 
PASS 
TESTING [2761.67e00] 
PASS 
TESTING [2761.67e000] 
PASS 
TESTING [7676546.67e-3] 
PASS 
TESTING [-0.03] 
PASS 
TESTING [-198.60] 
PASS 
TESTING [-2000045.178] 
PASS 
TESTING [-1.7976931348623157e+308] 
PASS 
TESTING [-0.000000000000000000890] 
PASS 
TESTING [-257953786.9864236576] 
PASS 
TESTING [-257953786.9864236576e8] 
PASS 
TESTING [-36.912e3] 
PASS 
TESTING [-2761.67e0] 
PASS 
TESTING [-2761.67e00] 
PASS 
TESTING [-2761.67e000] 
PASS 
TESTING [-7676546.67e-3] 
PASS 
--- PASS: TestMixedModeFixEncodedInt (0.01s)
=== RUN   TestCodecDesc
--- PASS: TestCodecDesc (0.00s)
=== RUN   TestCodecDescPropLen
--- PASS: TestCodecDescPropLen (0.00s)
=== RUN   TestCodecDescSplChar
--- PASS: TestCodecDescSplChar (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/collatejson	0.035s
Initializing write barrier = 8000
=== RUN   TestForestDBIterator
2022-09-03T02:56:03.649+05:30 [INFO][FDB] Forestdb blockcache size 134217728 initialized in 5021 us

2022-09-03T02:56:03.650+05:30 [INFO][FDB] Forestdb opened database file test
2022-09-03T02:56:03.655+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIterator (0.01s)
=== RUN   TestForestDBIteratorSeek
2022-09-03T02:56:03.656+05:30 [INFO][FDB] Forestdb opened database file test
2022-09-03T02:56:03.662+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIteratorSeek (0.01s)
=== RUN   TestPrimaryIndexEntry
--- PASS: TestPrimaryIndexEntry (0.00s)
=== RUN   TestSecondaryIndexEntry
--- PASS: TestSecondaryIndexEntry (0.00s)
=== RUN   TestPrimaryIndexEntryMatch
--- PASS: TestPrimaryIndexEntryMatch (0.00s)
=== RUN   TestSecondaryIndexEntryMatch
--- PASS: TestSecondaryIndexEntryMatch (0.00s)
=== RUN   TestLongDocIdEntry
--- PASS: TestLongDocIdEntry (0.00s)
=== RUN   TestMemDBInsertionPerf
Maximum number of file descriptors = 200000
Set IO Concurrency: 7200
Initial build: 10000000 items took 44.317494554s -> 225644.52482337863 items/s
Incr build: 10000000 items took 3m6.067237692s -> 53744.01277753774 items/s
Main Index: {
"node_count":             18000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       3,
"next_pointers_per_node": 1.3333,
"memory_used":            1695887724,
"node_allocs":            18000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 13500190,
"level1": 3375549,
"level2": 843074,
"level3": 210794,
"level4": 52734,
"level5": 13349,
"level6": 3244,
"level7": 782,
"level8": 221,
"level9": 42,
"level10": 17,
"level11": 3,
"level12": 1,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Back Index 0 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 1 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 2 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 3 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 4 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 5 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 6 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 7 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 8 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 9 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 10 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 11 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 12 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 13 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 14 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 15 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
--- PASS: TestMemDBInsertionPerf (230.39s)
=== RUN   TestBasicsA
--- PASS: TestBasicsA (0.00s)
=== RUN   TestSizeA
--- PASS: TestSizeA (0.00s)
=== RUN   TestSizeWithFreelistA
--- PASS: TestSizeWithFreelistA (0.00s)
=== RUN   TestDequeueUptoSeqnoA
--- PASS: TestDequeueUptoSeqnoA (0.10s)
=== RUN   TestDequeueA
--- PASS: TestDequeueA (1.21s)
=== RUN   TestMultipleVbucketsA
--- PASS: TestMultipleVbucketsA (0.00s)
=== RUN   TestDequeueUptoFreelistA
--- PASS: TestDequeueUptoFreelistA (0.00s)
=== RUN   TestDequeueUptoFreelistMultVbA
--- PASS: TestDequeueUptoFreelistMultVbA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA
--- PASS: TestConcurrentEnqueueDequeueA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA1
--- PASS: TestConcurrentEnqueueDequeueA1 (10.01s)
=== RUN   TestEnqueueAppCh
--- PASS: TestEnqueueAppCh (2.00s)
=== RUN   TestDequeueN
--- PASS: TestDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN
--- PASS: TestConcurrentEnqueueDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN1
--- PASS: TestConcurrentEnqueueDequeueN1 (10.01s)
PASS
ok  	github.com/couchbase/indexing/secondary/indexer	254.452s
=== RUN   TestConnPoolBasicSanity
2022-09-03T03:00:21.446+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 3 overflow 6 low WM 3 relConn batch size 1 ...
2022-09-03T03:00:21.653+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-03T03:00:22.447+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnPoolBasicSanity (5.00s)
=== RUN   TestConnRelease
2022-09-03T03:00:26.450+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Waiting for connections to get released
Waiting for more connections to get released
Waiting for further more connections to get released
2022-09-03T03:01:06.205+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-03T03:01:06.469+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnRelease (43.76s)
=== RUN   TestLongevity
2022-09-03T03:01:10.206+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Releasing 1 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 4 conns.
Getting 1 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 3 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 2 conns.
Getting 1 conns.
Releasing 0 conns.
Getting 1 conns.
Releasing 3 conns.
Getting 3 conns.
Releasing 2 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
2022-09-03T03:01:48.602+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-03T03:01:49.225+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLongevity (42.40s)
=== RUN   TestSustainedHighConns
2022-09-03T03:01:52.603+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Returning 1 Connections
Allocating 8 Connections
cp.curActConns = 0
cp.curActConns = 7
Returning 2 Connections
Allocating 6 Connections
cp.curActConns = 11
Returning 4 Connections
Allocating 6 Connections
cp.curActConns = 11
Returning 2 Connections
cp.curActConns = 11
Allocating 6 Connections
Returning 0 Connections
cp.curActConns = 17
Returning 1 Connections
Allocating 13 Connections
cp.curActConns = 22
Returning 0 Connections
cp.curActConns = 29
Returning 4 Connections
Allocating 13 Connections
cp.curActConns = 34
Returning 3 Connections
cp.curActConns = 35
Allocating 22 Connections
Returning 3 Connections
cp.curActConns = 45
Returning 4 Connections
cp.curActConns = 50
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 49
Returning 2 Connections
Allocating 21 Connections
cp.curActConns = 56
Returning 4 Connections
cp.curActConns = 64
Returning 4 Connections
Allocating 0 Connections
cp.curActConns = 60
Returning 0 Connections
Allocating 13 Connections
cp.curActConns = 67
Returning 3 Connections
cp.curActConns = 70
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 72
Returning 1 Connections
Allocating 10 Connections
cp.curActConns = 79
Returning 0 Connections
cp.curActConns = 81
Returning 1 Connections
Allocating 16 Connections
cp.curActConns = 91
Returning 3 Connections
cp.curActConns = 93
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 94
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 103
Returning 3 Connections
cp.curActConns = 109
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 110
Returning 3 Connections
Allocating 22 Connections
cp.curActConns = 115
Returning 4 Connections
cp.curActConns = 125
Returning 2 Connections
Allocating 13 Connections
cp.curActConns = 126
Returning 1 Connections
cp.curActConns = 135
Returning 0 Connections
Allocating 23 Connections
cp.curActConns = 137
Returning 3 Connections
cp.curActConns = 148
Returning 2 Connections
cp.curActConns = 153
Returning 3 Connections
Allocating 16 Connections
cp.curActConns = 159
Returning 4 Connections
cp.curActConns = 162
Returning 3 Connections
Allocating 18 Connections
cp.curActConns = 170
Returning 1 Connections
cp.curActConns = 176
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 177
Returning 1 Connections
Allocating 21 Connections
cp.curActConns = 181
Returning 2 Connections
cp.curActConns = 192
Returning 0 Connections
cp.curActConns = 195
Returning 3 Connections
Allocating 2 Connections
cp.curActConns = 194
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 196
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 193
Returning 1 Connections
Allocating 15 Connections
cp.curActConns = 204
Returning 2 Connections
cp.curActConns = 205
Allocating 2 Connections
Returning 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 2 Connections
cp.curActConns = 209
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 209
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 210
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 208
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 208
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 212
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 215
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 214
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 212
Returning 4 Connections
Allocating 1 Connections
cp.curActConns = 209
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 209
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 208
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 1 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 211
Returning 4 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 213
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 213
Returning 3 Connections
Allocating 4 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 215
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 215
Returning 3 Connections
cp.curActConns = 213
Allocating 3 Connections
Returning 1 Connections
cp.curActConns = 215
Allocating 0 Connections
Returning 0 Connections
cp.curActConns = 215
Allocating 0 Connections
Returning 4 Connections
cp.curActConns = 211
Allocating 2 Connections
Returning 2 Connections
cp.curActConns = 211
Allocating 0 Connections
Returning 1 Connections
cp.curActConns = 210
Allocating 4 Connections
Returning 0 Connections
cp.curActConns = 214
Allocating 3 Connections
Returning 0 Connections
cp.curActConns = 217
Allocating 4 Connections
Returning 0 Connections
cp.curActConns = 221
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 218
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 216
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 216
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 218
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 221
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 220
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 219
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 222
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 221
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 220
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 224
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 224
Retuning from startDeallocatorRoutine
Retuning from startAllocatorRoutine
2022-09-03T03:02:47.663+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-03T03:02:48.637+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestSustainedHighConns (59.06s)
=== RUN   TestLowWM
2022-09-03T03:02:51.664+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 20 overflow 5 low WM 10 relConn batch size 2 ...
2022-09-03T03:03:51.680+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-09-03T03:04:51.696+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-09-03T03:04:57.174+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-03T03:04:57.697+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLowWM (129.51s)
=== RUN   TestTotalConns
2022-09-03T03:05:01.176+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 120 overflow 5 low WM 10 relConn batch size 10 ...
2022-09-03T03:05:15.357+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-03T03:05:16.182+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestTotalConns (18.18s)
=== RUN   TestUpdateTickRate
2022-09-03T03:05:19.359+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 40 overflow 5 low WM 2 relConn batch size 2 ...
2022-09-03T03:05:40.208+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-03T03:05:40.370+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestUpdateTickRate (24.85s)
PASS
ok  	github.com/couchbase/indexing/secondary/queryport/client	322.811s
Starting server: attempt 1

Functional tests

2022/09/03 03:07:52 In TestMain()
2022/09/03 03:07:52 otp node fetch error: json: cannot unmarshal string into Go value of type couchbase.Pool
2022/09/03 03:07:52 Initialising services with role: kv,n1ql on node: 127.0.0.1:9000
2022/09/03 03:07:53 Initialising web UI on node: 127.0.0.1:9000
2022/09/03 03:07:53 InitWebCreds, response is: {"newBaseUri":"http://127.0.0.1:9000/"}
2022/09/03 03:07:54 Setting data quota of 1500M and Index quota of 1500M
2022/09/03 03:07:55 Adding node: https://127.0.0.1:19001 with role: kv,index to the cluster
2022/09/03 03:08:01 AddNode: Successfully added node: 127.0.0.1:9001 (role kv,index), response: {"otpNode":"n_1@127.0.0.1"}
2022/09/03 03:08:06 Rebalance progress: 0
2022/09/03 03:08:11 Rebalance progress: 0
2022/09/03 03:08:16 Rebalance progress: 100
2022/09/03 03:08:21 Created bucket default, responseBody: 
2022/09/03 03:08:26 Cluster status: map[127.0.0.1:9001:[index kv] 172.31.5.112:9000:[kv n1ql]]
2022/09/03 03:08:26 Successfully initialised cluster
2022/09/03 03:08:26 Cluster status: map[127.0.0.1:9001:[index kv] 172.31.5.112:9000:[kv n1ql]]
2022-09-03T03:08:26.875+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = EOF. Kill Pipe.
2022-09-03T03:08:26.875+05:30 [Error] FollowerSyncProxy.receiveAndUpdateAcceptedEpoch(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2022-09-03T03:08:26.875+05:30 [Error] WatcherServer.runOnce() : Watcher fail to synchronized with peer 127.0.0.1:9106
2022/09/03 03:08:27 ChangeIndexerSettings: Host  Port 0 Nodes []
2022/09/03 03:08:27 Changing config key queryport.client.settings.backfillLimit to value 0
2022/09/03 03:08:27 Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused
panic: Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused


goroutine 1 [running]:
panic({0xe79d00, 0xc00010fd00})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/panic.go:941 +0x397 fp=0xc0000e9ca8 sp=0xc0000e9be8 pc=0x43a6b7
log.Panicf({0x10142f0?, 0x1e?}, {0xc0000e9df8?, 0x27?, 0xe6f2c0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/log/log.go:392 +0x67 fp=0xc0000e9cf0 sp=0xc0000e9ca8 pc=0x5bcf27
github.com/couchbase/indexing/secondary/tests/framework/common.HandleError(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/framework/common/util.go:76
github.com/couchbase/indexing/secondary/tests/functionaltests.TestMain(0x445111?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/functionaltests/common_test.go:104 +0x5c7 fp=0xc0000e9ec8 sp=0xc0000e9cf0 pc=0xd20567
main.main()
	_testmain.go:483 +0x1d3 fp=0xc0000e9f80 sp=0xc0000e9ec8 pc=0xdfb953
runtime.main()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:250 +0x212 fp=0xc0000e9fe0 sp=0xc0000e9f80 pc=0x43d232
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000e9fe8 sp=0xc0000e9fe0 pc=0x46db81

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060fb0 sp=0xc000060f90 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.forcegchelper()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:301 +0xad fp=0xc000060fe0 sp=0xc000060fb0 pc=0x43d48d
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x46db81
created by runtime.init.6
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:289 +0x25

goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005c790 sp=0xc00005c770 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgsweep(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005c7c8 sp=0xc00005c790 pc=0x428757
runtime.gcenable.func1()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x26 fp=0xc00005c7e0 sp=0xc00005c7c8 pc=0x41e306
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46db81
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x6b

goroutine 19 [GC scavenge wait]:
runtime.gopark(0x39b3d33f85a6?, 0x10000?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005cf20 sp=0xc00005cf00 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgscavenge(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc00005cfc8 sp=0xc00005cf20 pc=0x426565
runtime.gcenable.func2()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0x26 fp=0xc00005cfe0 sp=0xc00005cfc8 pc=0x41e2a6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46db81
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0xaa

goroutine 20 [finalizer wait]:
runtime.gopark(0x0?, 0x108e058?, 0x60?, 0x40?, 0x2000000020?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060630 sp=0xc000060610 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.runfinq()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000607e0 sp=0xc000060630 pc=0x41d3b3
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x46db81
created by runtime.createfing
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:157 +0x45

goroutine 34 [select]:
runtime.gopark(0xc00005d798?, 0x2?, 0xa7?, 0xd6?, 0xc00005d78c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005d618 sp=0xc00005d5f8 pc=0x43d5f6
runtime.selectgo(0xc00005d798, 0xc00005d788, 0x0?, 0x0, 0xb?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc00005d758 sp=0xc00005d618 pc=0x44d072
github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0xc00022c0d8)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:396 +0x67 fp=0xc00005d7c8 sp=0xc00005d758 pc=0x784d67
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func2()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x26 fp=0xc00005d7e0 sp=0xc00005d7c8 pc=0x7859e6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x46db81
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x37a

goroutine 35 [select]:
runtime.gopark(0xc000248798?, 0x2?, 0x0?, 0x0?, 0xc00024878c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000248608 sp=0xc0002485e8 pc=0x43d5f6
runtime.selectgo(0xc000248798, 0xc000248788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000248748 sp=0xc000248608 pc=0x44d072
github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0xc00022c0f0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:316 +0x85 fp=0xc0002487c8 sp=0xc000248748 pc=0x784785
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func3()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x26 fp=0xc0002487e0 sp=0xc0002487c8 pc=0x785986
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002487e8 sp=0xc0002487e0 pc=0x46db81
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x3ca

goroutine 36 [IO wait]:
runtime.gopark(0xc00022e340?, 0xc000052a00?, 0x70?, 0xf8?, 0x4834a2?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004bf800 sp=0xc0004bf7e0 pc=0x43d5f6
runtime.netpollblock(0xc0000a9000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc0004bf838 sp=0xc0004bf800 pc=0x436097
internal/poll.runtime_pollWait(0x7fd251f93fc8, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc0004bf858 sp=0xc0004bf838 pc=0x468169
internal/poll.(*pollDesc).wait(0xc000032200?, 0xc0000a9000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc0004bf880 sp=0xc0004bf858 pc=0x4a1092
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000032200, {0xc0000a9000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0004bf900 sp=0xc0004bf880 pc=0x4a23fa
net.(*netFD).Read(0xc000032200, {0xc0000a9000?, 0xc0004b6c50?, 0xc0004bf9d0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004bf948 sp=0xc0004bf900 pc=0x668169
net.(*conn).Read(0xc000010028, {0xc0000a9000?, 0x11?, 0xc0004bfa68?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc0004bf990 sp=0xc0004bf948 pc=0x6783e5
bufio.(*Reader).Read(0xc000094120, {0xc000038601, 0x5ff, 0x452894?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:236 +0x1b4 fp=0xc0004bf9c8 sp=0xc0004bf990 pc=0x51f654
github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x191?, {0xc000038601?, 0x8?, 0x20?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:102 +0x25 fp=0xc0004bf9f8 sp=0xc0004bf9c8 pc=0x7d5fa5
encoding/json.(*Decoder).refill(0xc0000b6000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc0004bfa48 sp=0xc0004bf9f8 pc=0x564b1f
encoding/json.(*Decoder).readValue(0xc0000b6000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc0004bfa98 sp=0xc0004bfa48 pc=0x56471b
encoding/json.(*Decoder).Decode(0xc0000b6000, {0xeaa960, 0xc000094200})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc0004bfac8 sp=0xc0004bfa98 pc=0x564378
net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc0000941e0, 0xc00004c100)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/jsonrpc/server.go:66 +0x85 fp=0xc0004bfb08 sp=0xc0004bfac8 pc=0x7d5205
github.com/couchbase/cbauth/revrpc.(*jsonServerCodec).ReadRequestHeader(0xc0000b4000?, 0x4cc2e8?)
	:1 +0x2a fp=0xc0004bfb28 sp=0xc0004bfb08 pc=0x7d7d2a
net/rpc.(*Server).readRequestHeader(0xc0000b4000, {0x11a26e8, 0xc000076160})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:587 +0x66 fp=0xc0004bfbf8 sp=0xc0004bfb28 pc=0x7d4926
net/rpc.(*Server).readRequest(0x0?, {0x11a26e8, 0xc000076160})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:547 +0x3b fp=0xc0004bfcd0 sp=0xc0004bfbf8 pc=0x7d447b
net/rpc.(*Server).ServeCodec(0xc0000b4000, {0x11a26e8?, 0xc000076160})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:462 +0x87 fp=0xc0004bfdc8 sp=0xc0004bfcd0 pc=0x7d3ba7
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc000248f60?, 0xc000073fa0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:192 +0x5d9 fp=0xc0004bff38 sp=0xc0004bfdc8 pc=0x7d66f9
github.com/couchbase/cbauth/revrpc.BabysitService(0x0?, 0x0?, {0x1198e00?, 0xc00000e030?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:288 +0x58 fp=0xc0004bff70 sp=0xc0004bff38 pc=0x7d6df8
github.com/couchbase/cbauth.runRPCForSvc(0x0?, 0xc000242000)
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xbd fp=0xc0004bffc0 sp=0xc0004bff70 pc=0x7e0b7d
github.com/couchbase/cbauth.startDefault.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x25 fp=0xc0004bffe0 sp=0xc0004bffc0 pc=0x7e0e65
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004bffe8 sp=0xc0004bffe0 pc=0x46db81
created by github.com/couchbase/cbauth.startDefault
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xf9

goroutine 13 [GC worker (idle)]:
runtime.gopark(0xc00013c520?, 0x0?, 0x0?, 0x83?, 0x11a2601?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000248f58 sp=0xc000248f38 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc000248fe0 sp=0xc000248f58 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000248fe8 sp=0xc000248fe0 pc=0x46db81
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 8 [select]:
runtime.gopark(0xc000075f90?, 0x2?, 0xd8?, 0x5d?, 0xc000075f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000075d90 sp=0xc000075d70 pc=0x43d5f6
runtime.selectgo(0xc000075f90, 0xc000075f20, 0xc000096300?, 0x0, 0xc000206090?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000075ed0 sp=0xc000075d90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc000172ea0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000075fc8 sp=0xc000075ed0 pc=0x76c115
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000075fe0 sp=0xc000075fc8 pc=0x768ba6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=0x46db81
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791

goroutine 7 [select]:
runtime.gopark(0xc000074f68?, 0x4?, 0x48?, 0x4c?, 0xc000074db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000074c00 sp=0xc000074be0 pc=0x43d5f6
runtime.selectgo(0xc000074f68, 0xc000074da8, 0xc000110140?, 0x0, 0xc000248d01?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000074d40 sp=0xc000074c00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc000172ea0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2213 +0xda5 fp=0xc000074fc8 sp=0xc000074d40 pc=0x76b025
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000074fe0 sp=0xc000074fc8 pc=0x768c06
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x46db81
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e

goroutine 43 [GC worker (idle)]:
runtime.gopark(0x1a7d760?, 0x1?, 0xa9?, 0x1c?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00024a758 sp=0xc00024a738 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00024a7e0 sp=0xc00024a758 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00024a7e8 sp=0xc00024a7e0 pc=0x46db81
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 42 [GC worker (idle)]:
runtime.nanotime(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/time_nofake.go:19
runtime.suspendG(0xc0000021a0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/preempt.go:229 +0x1a5 fp=0x7fd250f4bcd0 sp=0x7fd250f4bc60 pc=0x43b725
runtime.markroot.func1()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcmark.go:233 +0x8d fp=0x7fd250f4bd20 sp=0x7fd250f4bcd0 pc=0x421c8d
runtime.markroot(0xc000053c38, 0x14, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcmark.go:214 +0x1a5 fp=0x7fd250f4bdc0 sp=0x7fd250f4bd20 pc=0x421945
runtime.gcDrain(0xc000053c38, 0x7)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcmark.go:1047 +0x39f fp=0x7fd250f4be20 sp=0x7fd250f4bdc0 pc=0x42395f
runtime.gcBgMarkWorker.func2()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1295 +0x6e fp=0x7fd250f4be70 sp=0x7fd250f4be20 pc=0x4207ae
runtime.systemstack()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:469 +0x49 fp=0x7fd250f4be78 sp=0x7fd250f4be70 pc=0x46b9c9
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 46 [chan receive]:
runtime.gopark(0x3?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000247678 sp=0xc000247658 pc=0x43d5f6
runtime.chanrecv(0xc0003f5260, 0xc000247790, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:577 +0x56c fp=0xc000247708 sp=0xc000247678 pc=0x40b5ac
runtime.chanrecv2(0xc00020d950?, 0x16?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:445 +0x18 fp=0xc000247730 sp=0xc000247708 pc=0x40b018
github.com/couchbase/goutils/systemeventlog.(*SystemEventLoggerImpl).logEvents(0xc00018c0e0)
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:186 +0xb7 fp=0xc0002477c8 sp=0xc000247730 pc=0xb79297
github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger.func1()
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x26 fp=0xc0002477e0 sp=0xc0002477c8 pc=0xb78ba6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002477e8 sp=0xc0002477e0 pc=0x46db81
created by github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x1d6

goroutine 26 [GC worker (idle)]:
runtime.gopark(0xe4c940?, 0xc00013c670?, 0x16?, 0x0?, 0x11a26e8?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000249758 sp=0xc000249738 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc0002497e0 sp=0xc000249758 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002497e8 sp=0xc0002497e0 pc=0x46db81
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 84 [select]:
runtime.gopark(0xc0005b2f68?, 0x4?, 0x3?, 0x0?, 0xc0005b2db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0005b2c00 sp=0xc0005b2be0 pc=0x43d5f6
runtime.selectgo(0xc0005b2f68, 0xc0005b2da8, 0xc000111b00?, 0x0, 0xc000605501?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0005b2d40 sp=0xc0005b2c00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc00027a360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2213 +0xda5 fp=0xc0005b2fc8 sp=0xc0005b2d40 pc=0x76b025
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc0005b2fe0 sp=0xc0005b2fc8 pc=0x768c06
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005b2fe8 sp=0xc0005b2fe0 pc=0x46db81
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e

goroutine 57 [select]:
runtime.gopark(0xc000602f90?, 0x2?, 0xf8?, 0xfc?, 0xc000602f8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000602e18 sp=0xc000602df8 pc=0x43d5f6
runtime.selectgo(0xc000602f90, 0xc000602f88, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000602f58 sp=0xc000602e18 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).listenMetaChange(0xc00012a400, 0xc0001aa3f0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1830 +0x70 fp=0xc000602fc0 sp=0xc000602f58 pc=0xcab870
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1823 +0x2a fp=0xc000602fe0 sp=0xc000602fc0 pc=0xcab7ca
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000602fe8 sp=0xc000602fe0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1823 +0x28a

goroutine 16 [select]:
runtime.gopark(0xc000244798?, 0x2?, 0x80?, 0x46?, 0xc00024478c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000244618 sp=0xc0002445f8 pc=0x43d5f6
runtime.selectgo(0xc000244798, 0xc000244788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000244758 sp=0xc000244618 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*schedTokenMonitor).updater(0xc0001143c0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2389 +0x92 fp=0xc0002447c8 sp=0xc000244758 pc=0xcbf192
github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x26 fp=0xc0002447e0 sp=0xc0002447c8 pc=0xcbdac6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002447e8 sp=0xc0002447e0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x2e5

goroutine 27 [select]:
runtime.gopark(0xc0004bbba0?, 0x3?, 0x0?, 0x30?, 0xc0004bbb1a?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004bb978 sp=0xc0004bb958 pc=0x43d5f6
runtime.selectgo(0xc0004bbba0, 0xc0004bbb14, 0x3?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0004bbab8 sp=0xc0004bb978 pc=0x44d072
github.com/couchbase/cbauth/metakv.doRunObserveChildren(0xc00020e7b0?, {0x1029dc0, 0x1b}, 0xc0004bbe68, 0xc0000469c0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:301 +0x429 fp=0xc0004bbe40 sp=0xc0004bbab8 pc=0x9c5e49
github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:259
github.com/couchbase/cbauth/metakv.RunObserveChildren({0x1029dc0?, 0x0?}, 0x0?, 0x0?)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:389 +0x58 fp=0xc0004bbe88 sp=0xc0004bbe40 pc=0x9c63f8
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0?, {0x0?, 0x0?})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1579 +0xc7 fp=0xc0004bbf00 sp=0xc0004bbe88 pc=0xb7e227
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc0004bbfa0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x83 fp=0xc0004bbf38 sp=0xc0004bbf00 pc=0xadd0c3
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1584 +0xdf fp=0xc0004bbfe0 sp=0xc0004bbf38 pc=0xb7e0df
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004bbfe8 sp=0xc0004bbfe0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1572 +0xaf

goroutine 69 [IO wait]:
runtime.gopark(0xc0000fb6c0?, 0xc00004e000?, 0x10?, 0xfa?, 0x4834a2?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0005af9a0 sp=0xc0005af980 pc=0x43d5f6
runtime.netpollblock(0xc0000d0000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc0005af9d8 sp=0xc0005af9a0 pc=0x436097
internal/poll.runtime_pollWait(0x7fd251f93ed8, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc0005af9f8 sp=0xc0005af9d8 pc=0x468169
internal/poll.(*pollDesc).wait(0xc00012a200?, 0xc0000d0000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc0005afa20 sp=0xc0005af9f8 pc=0x4a1092
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc00012a200, {0xc0000d0000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0005afaa0 sp=0xc0005afa20 pc=0x4a23fa
net.(*netFD).Read(0xc00012a200, {0xc0000d0000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc0005afae8 sp=0xc0005afaa0 pc=0x668169
net.(*conn).Read(0xc0000101e8, {0xc0000d0000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc0005afb30 sp=0xc0005afae8 pc=0x6783e5
net/http.(*persistConn).Read(0xc000172ea0, {0xc0000d0000?, 0x1000?, 0x1000?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc0005afb90 sp=0xc0005afb30 pc=0x76960e
bufio.(*Reader).fill(0xc0000943c0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc0005afbc8 sp=0xc0005afb90 pc=0x51f083
bufio.(*Reader).ReadSlice(0xc0000943c0, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:371 +0x2f fp=0xc0005afc18 sp=0xc0005afbc8 pc=0x51fc6f
net/http/internal.readChunkLine(0x400?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc0005afc68 sp=0xc0005afc18 pc=0x707445
net/http/internal.(*chunkedReader).beginChunk(0xc0002060f0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc0005afc98 sp=0xc0005afc68 pc=0x706ec8
net/http/internal.(*chunkedReader).Read(0xc0002060f0, {0xc00060c000?, 0x0?, 0xc000061d68?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc0005afd18 sp=0xc0005afc98 pc=0x70718e
net/http.(*body).readLocked(0xc000110140, {0xc00060c000?, 0x7fd27958abb8?, 0xc000061e35?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:844 +0x3c fp=0xc0005afd68 sp=0xc0005afd18 pc=0x75e17c
net/http.(*body).Read(0x1010000000000?, {0xc00060c000?, 0x0?, 0x7fd279581108?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:836 +0x125 fp=0xc0005afde0 sp=0xc0005afd68 pc=0x75e045
net/http.(*bodyEOFSignal).Read(0xc0001105c0, {0xc00060c000, 0x200, 0x200})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2774 +0x142 fp=0xc0005afe60 sp=0xc0005afde0 pc=0x76dd42
encoding/json.(*Decoder).refill(0xc0000b6140)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc0005afeb0 sp=0xc0005afe60 pc=0x564b1f
encoding/json.(*Decoder).readValue(0xc0000b6140)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc0005aff00 sp=0xc0005afeb0 pc=0x56471b
encoding/json.(*Decoder).Decode(0xc0000b6140, {0xe4f2c0, 0xc000218190})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc0005aff30 sp=0xc0005aff00 pc=0x564378
github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x10b fp=0xc0005affe0 sp=0xc0005aff30 pc=0x9c62eb
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005affe8 sp=0xc0005affe0 pc=0x46db81
created by github.com/couchbase/cbauth/metakv.doRunObserveChildren
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:280 +0x2eb

goroutine 73 [select]:
runtime.gopark(0xc0006076b8?, 0x2?, 0x6a?, 0x35?, 0xc000607690?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000607518 sp=0xc0006074f8 pc=0x43d5f6
runtime.selectgo(0xc0006076b8, 0xc00060768c, 0x119b780?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000607658 sp=0xc000607518 pc=0x44d072
github.com/couchbase/indexing/secondary/manager/client.(*watcher).waitForReady(0xc00060a300, 0xc000116fc0, 0xc00013d450?, 0xc0006077d0?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:5841 +0x239 fp=0xc000607720 sp=0xc000607658 pc=0xc8b319
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).retryHelper(0xc0000d6370, 0xe?, 0x119b780?, {0xc00013cc50, 0xe}, {0xc0000488c0, 0x1b}, 0x1a7c720?, 0xc0004b69b0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4089 +0x65 fp=0xc000607788 sp=0xc000607720 pc=0xc7f925
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).WatchMetadata.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:341 +0x46 fp=0xc0006077e0 sp=0xc000607788 pc=0xc689e6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0006077e8 sp=0xc0006077e0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).WatchMetadata
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:341 +0x4d4

goroutine 53 [IO wait]:
runtime.gopark(0xc0000fb860?, 0xc000054f00?, 0x60?, 0xb7?, 0x4834a2?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00015b6f0 sp=0xc00015b6d0 pc=0x43d5f6
runtime.netpollblock(0xc0004e9000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc00015b728 sp=0xc00015b6f0 pc=0x436097
internal/poll.runtime_pollWait(0x7fd251f93c08, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc00015b748 sp=0xc00015b728 pc=0x468169
internal/poll.(*pollDesc).wait(0xc00012a100?, 0xc0004e9000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc00015b770 sp=0xc00015b748 pc=0x4a1092
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc00012a100, {0xc0004e9000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00015b7f0 sp=0xc00015b770 pc=0x4a23fa
net.(*netFD).Read(0xc00012a100, {0xc0004e9000?, 0xc00015b8b8?, 0x552725?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc00015b838 sp=0xc00015b7f0 pc=0x668169
net.(*conn).Read(0xc0001ce020, {0xc0004e9000?, 0x198?, 0xe659c0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc00015b880 sp=0xc00015b838 pc=0x6783e5
net/http.(*persistConn).Read(0xc00027a360, {0xc0004e9000?, 0xc00015b910?, 0xc000634120?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc00015b8e0 sp=0xc00015b880 pc=0x76960e
bufio.(*Reader).fill(0xc000094840)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc00015b918 sp=0xc00015b8e0 pc=0x51f083
bufio.(*Reader).ReadSlice(0xc000094840, 0xbd?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:371 +0x2f fp=0xc00015b968 sp=0xc00015b918 pc=0x51fc6f
net/http/internal.readChunkLine(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc00015b9b8 sp=0xc00015b968 pc=0x707445
net/http/internal.(*chunkedReader).beginChunk(0xc000207020)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc00015b9e8 sp=0xc00015b9b8 pc=0x706ec8
net/http/internal.(*chunkedReader).Read(0xc000207020, {0xc000645000?, 0x0?, 0xf?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc00015ba68 sp=0xc00015b9e8 pc=0x70718e
net/http.(*body).readLocked(0xc000111b00, {0xc000645000?, 0x0?, 0x1?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:844 +0x3c fp=0xc00015bab8 sp=0xc00015ba68 pc=0x75e17c
net/http.(*body).Read(0xc0000c8f84?, {0xc000645000?, 0xc0000931d0?, 0xc00017ad80?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:836 +0x125 fp=0xc00015bb30 sp=0xc00015bab8 pc=0x75e045
net/http.(*bodyEOFSignal).Read(0xc000111b40, {0xc000645000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2774 +0x142 fp=0xc00015bbb0 sp=0xc00015bb30 pc=0x76dd42
bufio.(*Reader).fill(0xc00015bf58)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc00015bbe8 sp=0xc00015bbb0 pc=0x51f083
bufio.(*Reader).ReadSlice(0xc00015bf58, 0x65?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:371 +0x2f fp=0xc00015bc38 sp=0xc00015bbe8 pc=0x51fc6f
bufio.(*Reader).collectFragments(0xf7f740?, 0x88?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:446 +0x74 fp=0xc00015bce8 sp=0xc00015bc38 pc=0x5200b4
bufio.(*Reader).ReadBytes(0xc000112840?, 0xbf?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:474 +0x1d fp=0xc00015bd68 sp=0xc00015bce8 pc=0x52025d
github.com/couchbase/indexing/secondary/common.(*internalVersionMonitor).waitForChange(0xc000146c30?, 0x2c?, 0x1044256?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:620 +0x4b fp=0xc00015be18 sp=0xc00015bd68 pc=0xad9beb
github.com/couchbase/indexing/secondary/common.(*internalVersionMonitor).notifier(0xc000220070)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:709 +0x493 fp=0xc00015bfc8 sp=0xc00015be18 pc=0xada493
github.com/couchbase/indexing/secondary/common.newInternalVersionMonitor.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:232 +0x26 fp=0xc00015bfe0 sp=0xc00015bfc8 pc=0xad5e06
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00015bfe8 sp=0xc00015bfe0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/common.newInternalVersionMonitor
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:232 +0x336

goroutine 72 [select]:
runtime.gopark(0xc00015df38?, 0x2?, 0x0?, 0x30?, 0xc00015df24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00015dd58 sp=0xc00015dd38 pc=0x43d5f6
runtime.selectgo(0xc00015df38, 0xc00015df20, 0xc000240310?, 0x0, 0xc00013d450?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc00015de98 sp=0xc00015dd58 pc=0x44d072
github.com/couchbase/gometa/protocol.RunWatcherServerWithRequest2({0xc00013cc50, 0xe}, {0x119b780, 0xc00060a300}, {0x11a81f8, 0xc00060a300}, {0x11a76e0, 0x1a7c720}, 0xc0002401c0, 0xc000116fc0, ...)
	/opt/build/goproj/src/github.com/couchbase/gometa/protocol/watcherServer.go:72 +0x20e fp=0xc00015df68 sp=0xc00015de98 pc=0xc1ef8e
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).startWatcher.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4155 +0x9f fp=0xc00015dfe0 sp=0xc00015df68 pc=0xc8011f
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00015dfe8 sp=0xc00015dfe0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).startWatcher
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4155 +0x1f8

goroutine 55 [select]:
runtime.gopark(0xc0004bcea8?, 0x6?, 0x68?, 0xcb?, 0xc0004bccbc?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004bcb20 sp=0xc0004bcb00 pc=0x43d5f6
runtime.selectgo(0xc0004bcea8, 0xc0004bccb0, 0x101b755?, 0x0, 0xc0004bcc98?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0004bcc60 sp=0xc0004bcb20 pc=0x44d072
net/http.(*persistConn).roundTrip(0xc00009c240, 0xc000111bc0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2620 +0x974 fp=0xc0004bcf18 sp=0xc0004bcc60 pc=0x76cfd4
net/http.(*Transport).roundTrip(0x19c4900, 0xc0000d2f00)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:594 +0x7c9 fp=0xc0004bd150 sp=0xc0004bcf18 pc=0x760a69
net/http.(*Transport).RoundTrip(0xc0000d2f00?, 0x1198580?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/roundtrip.go:17 +0x19 fp=0xc0004bd170 sp=0xc0004bd150 pc=0x748c99
net/http.send(0xc0000d2e00, {0x1198580, 0x19c4900}, {0xfa0260?, 0x48e901?, 0x1a460c0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:252 +0x5d8 fp=0xc0004bd350 sp=0xc0004bd170 pc=0x70a598
net/http.(*Client).send(0xc000207230, 0xc0000d2e00, {0x7fd27958d4a0?, 0x150?, 0x1a460c0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:176 +0x9b fp=0xc0004bd3c8 sp=0xc0004bd350 pc=0x709e3b
net/http.(*Client).do(0xc000207230, 0xc0000d2e00)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:725 +0x8f5 fp=0xc0004bd5c8 sp=0xc0004bd3c8 pc=0x70c275
net/http.(*Client).Do(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:593
github.com/couchbase/indexing/secondary/security.getWithAuthInternal({0xc000210b40?, 0x1b?}, 0xc0004bd960, {0x0, 0x0}, 0x0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:669 +0x549 fp=0xc0004bd6c8 sp=0xc0004bd5c8 pc=0x7e5e69
github.com/couchbase/indexing/secondary/security.GetWithAuthNonTLS(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:604
github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0xc0005aaf30, {0x101383c, 0x6}, {0xefd8a0?, 0xc00006fa01?}, {0xe50c00, 0xc000630398}, 0xc0001171a0?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:359 +0x1b3 fp=0xc0004bd910 sp=0xc0004bd6c8 pc=0x808753
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:530
github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth({0xc0001171a0, 0x3f}, {0x1197f00?, 0xc00046a900})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:594 +0x125 fp=0xc0004bd988 sp=0xc0004bd910 pc=0x80a705
github.com/couchbase/indexing/secondary/dcp.Connect({0xc0001171a0, 0x3f})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:600 +0xce fp=0xc0004bdb20 sp=0xc0004bd988 pc=0x80a8ae
github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier({0xc0001171a0, 0x3f}, {0x1014910, 0x7}, {0x10184e7, 0xb})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:239 +0x1a8 fp=0xc0004bdda0 sp=0xc0004bdb20 pc=0xadf9c8
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).watchClusterChanges(0xc0000d62c0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2071 +0xff fp=0xc0004bdfc8 sp=0xc0004bdda0 pc=0xcbce9f
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x26 fp=0xc0004bdfe0 sp=0xc0004bdfc8 pc=0xcb0626
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004bdfe8 sp=0xc0004bdfe0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x5c5

goroutine 85 [select]:
runtime.gopark(0xc0005b3f90?, 0x2?, 0xd8?, 0x3d?, 0xc0005b3f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0005b3d90 sp=0xc0005b3d70 pc=0x43d5f6
runtime.selectgo(0xc0005b3f90, 0xc0005b3f20, 0xc0001d20c0?, 0x0, 0xc0006015c0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0005b3ed0 sp=0xc0005b3d90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc00027a360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc0005b3fc8 sp=0xc0005b3ed0 pc=0x76c115
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc0005b3fe0 sp=0xc0005b3fc8 pc=0x768ba6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005b3fe8 sp=0xc0005b3fe0 pc=0x46db81
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791

goroutine 70 [runnable]:
runtime.gopark(0xc000102d00?, 0xc000054f00?, 0x68?, 0xb?, 0x4834a2?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0005b0af8 sp=0xc0005b0ad8 pc=0x43d5f6
runtime.netpollblock(0xc00060e000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc0005b0b30 sp=0xc0005b0af8 pc=0x436097
internal/poll.runtime_pollWait(0x7fd251f93b18, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc0005b0b50 sp=0xc0005b0b30 pc=0x468169
internal/poll.(*pollDesc).wait(0xc00024c080?, 0xc00060e000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc0005b0b78 sp=0xc0005b0b50 pc=0x4a1092
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc00024c080, {0xc00060e000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0005b0bf8 sp=0xc0005b0b78 pc=0x4a23fa
net.(*netFD).Read(0xc00024c080, {0xc00060e000?, 0x40aa49?, 0x4?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc0005b0c40 sp=0xc0005b0bf8 pc=0x668169
net.(*conn).Read(0xc000212018, {0xc00060e000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc0005b0c88 sp=0xc0005b0c40 pc=0x6783e5
net/http.(*persistConn).Read(0xc00009c240, {0xc00060e000?, 0xc0001166c0?, 0xc0005b0d30?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc0005b0ce8 sp=0xc0005b0c88 pc=0x76960e
bufio.(*Reader).fill(0xc0003f4ea0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc0005b0d20 sp=0xc0005b0ce8 pc=0x51f083
bufio.(*Reader).Peek(0xc0003f4ea0, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:144 +0x5d fp=0xc0005b0d40 sp=0xc0005b0d20 pc=0x51f1dd
net/http.(*persistConn).readLoop(0xc00009c240)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2093 +0x1ac fp=0xc0005b0fc8 sp=0xc0005b0d40 pc=0x76a42c
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc0005b0fe0 sp=0xc0005b0fc8 pc=0x768c06
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005b0fe8 sp=0xc0005b0fe0 pc=0x46db81
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e

goroutine 71 [select]:
runtime.gopark(0xc000073f90?, 0x2?, 0xd8?, 0x3d?, 0xc000073f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000073d90 sp=0xc000073d70 pc=0x43d5f6
runtime.selectgo(0xc000073f90, 0xc000073f20, 0xc000111600?, 0x0, 0xc0002072c0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000073ed0 sp=0xc000073d90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc00009c240)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000073fc8 sp=0xc000073ed0 pc=0x76c115
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x768ba6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x46db81
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791

goroutine 56 [chan receive]:
runtime.gopark(0xc000218328?, 0xc0000fb380?, 0x20?, 0x37?, 0x459da5?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0006036c8 sp=0xc0006036a8 pc=0x43d5f6
runtime.chanrecv(0xc0003f5b60, 0x0, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:577 +0x56c fp=0xc000603758 sp=0xc0006036c8 pc=0x40b5ac
runtime.chanrecv1(0xdf8475800?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:440 +0x18 fp=0xc000603780 sp=0xc000603758 pc=0x40afd8
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).logstats(0xc0000d62c0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:1410 +0x79 fp=0xc0006037c8 sp=0xc000603780 pc=0xcb7c39
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x26 fp=0xc0006037e0 sp=0xc0006037c8 pc=0xcb05c6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0006037e8 sp=0xc0006037e0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x605

goroutine 58 [select]:
runtime.gopark(0xc000603f50?, 0x2?, 0x0?, 0x30?, 0xc000603f14?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000603d90 sp=0xc000603d70 pc=0x43d5f6
runtime.selectgo(0xc000603f50, 0xc000603f10, 0x1052eb5?, 0x0, 0xe79200?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000603ed0 sp=0xc000603d90 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).logstats(0xc00012a400, 0xc0001aa3f0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1849 +0x238 fp=0xc000603fc0 sp=0xc000603ed0 pc=0xcabaf8
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1824 +0x2a fp=0xc000603fe0 sp=0xc000603fc0 pc=0xcab76a
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000603fe8 sp=0xc000603fe0 pc=0x46db81
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1824 +0x2ed
signal: aborted (core dumped)
FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests	35.897s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 52902    0 52902    0     0  1228k      0 --:--:-- --:--:-- --:--:-- 1260k
2022/09/03 03:08:30 In TestMain()
2022/09/03 03:08:30 Changing config key indexer.api.enableTestServer to value true
2022/09/03 03:08:30 Using memory_optimized for creating indexes
2022/09/03 03:08:30 Changing config key indexer.settings.storage_mode to value memory_optimized
=== RUN   TestRangeWithConcurrentAddMuts
2022/09/03 03:08:35 In TestRangeWithConcurrentAddMuts()
2022/09/03 03:08:35 In DropAllSecondaryIndexes()
2022/09/03 03:08:35 Generating JSON docs
2022/09/03 03:08:35 Setting initial JSON docs in KV
2022/09/03 03:08:36 All indexers are active
2022/09/03 03:08:36 Creating a 2i
2022/09/03 03:08:39 Created the secondary index index_company. Waiting for it become active
2022/09/03 03:08:39 Index is 10298590940485192228 now active
2022/09/03 03:08:39 In Range Scan for Thread 1: 
2022/09/03 03:08:39 CreateDocs:: Creating mutations
2022/09/03 03:08:39 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
--- PASS: TestRangeWithConcurrentAddMuts (124.18s)
=== RUN   TestRangeWithConcurrentDelMuts
2022-09-03T03:10:39.733+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:50550->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/09/03 03:10:39 In TestRangeWithConcurrentDelMuts()
2022/09/03 03:10:39 Generating JSON docs
2022/09/03 03:10:41 Setting initial JSON docs in KV
2022/09/03 03:11:03 All indexers are active
2022/09/03 03:11:03 Creating a 2i
2022/09/03 03:11:03 Index found:  index_company
2022/09/03 03:11:03 CreateDocs:: Delete mutations
2022/09/03 03:11:03 In Range Scan for Thread 1: 
2022/09/03 03:11:03 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
--- PASS: TestRangeWithConcurrentDelMuts (143.47s)
=== RUN   TestScanWithConcurrentIndexOps
2022/09/03 03:13:03 In TestScanWithConcurrentIndexOps()
2022/09/03 03:13:03 Generating JSON docs
2022-09-03T03:13:03.203+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:53624->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/09/03 03:13:08 Setting initial JSON docs in KV
2022/09/03 03:14:19 All indexers are active
2022/09/03 03:14:19 Creating a 2i
2022/09/03 03:14:19 Index found:  index_company
2022/09/03 03:14:19 In Range Scan for Thread 1: 
2022/09/03 03:14:19 Create and Drop index operations
2022/09/03 03:14:19 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
2022/09/03 03:14:37 Created the secondary index index_age. Waiting for it become active
2022/09/03 03:14:37 Index is 10682220616772222577 now active
2022/09/03 03:14:54 Created the secondary index index_firstname. Waiting for it become active
2022/09/03 03:14:54 Index is 3062167430278770360 now active
2022/09/03 03:14:55 Dropping the secondary index index_age
2022/09/03 03:14:56 Index dropped
2022/09/03 03:14:57 Dropping the secondary index index_firstname
2022/09/03 03:14:57 Index dropped
2022/09/03 03:15:14 Created the secondary index index_age. Waiting for it become active
2022/09/03 03:15:14 Index is 13478044166739039101 now active
2022/09/03 03:15:30 Created the secondary index index_firstname. Waiting for it become active
2022/09/03 03:15:30 Index is 9248530263697254805 now active
2022/09/03 03:15:32 Dropping the secondary index index_age
2022/09/03 03:15:32 Index dropped
2022/09/03 03:15:33 Dropping the secondary index index_firstname
2022/09/03 03:15:33 Index dropped
2022/09/03 03:15:50 Created the secondary index index_age. Waiting for it become active
2022/09/03 03:15:50 Index is 13272143871985553649 now active
2022/09/03 03:16:07 Created the secondary index index_firstname. Waiting for it become active
2022/09/03 03:16:07 Index is 8846965373131001934 now active
2022/09/03 03:16:09 Dropping the secondary index index_age
2022/09/03 03:16:09 Index dropped
2022/09/03 03:16:10 Dropping the secondary index index_firstname
2022/09/03 03:16:10 Index dropped
2022-09-03T03:16:19.327+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:58906->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/09/03 03:16:25 Created the secondary index index_age. Waiting for it become active
2022/09/03 03:16:25 Index is 17812421202264141015 now active
2022/09/03 03:16:38 Created the secondary index index_firstname. Waiting for it become active
2022/09/03 03:16:38 Index is 6113949926029259084 now active
2022/09/03 03:16:39 Dropping the secondary index index_age
2022/09/03 03:16:39 Index dropped
2022/09/03 03:16:40 Dropping the secondary index index_firstname
2022/09/03 03:16:41 Index dropped
--- PASS: TestScanWithConcurrentIndexOps (218.81s)
=== RUN   TestConcurrentScans_SameIndex
2022/09/03 03:16:42 In TestConcurrentScans_SameIndex()
2022/09/03 03:16:42 Generating JSON docs
2022-09-03T03:16:42.014+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:58908->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/09/03 03:16:46 Setting initial JSON docs in KV
2022/09/03 03:17:59 All indexers are active
2022/09/03 03:17:59 Creating a 2i
2022/09/03 03:17:59 Index found:  index_company
2022/09/03 03:17:59 In Range Scan for Thread 6: 
2022/09/03 03:17:59 In Range Scan for Thread 3: 
2022/09/03 03:17:59 In Range Scan for Thread 4: 
2022/09/03 03:17:59 In Range Scan for Thread 1: 
2022/09/03 03:17:59 In Range Scan for Thread 5: 
2022/09/03 03:17:59 In Range Scan for Thread 2: 
2022/09/03 03:17:59 ListAllSecondaryIndexes() for Thread 6: : Index index_company Bucket default
2022/09/03 03:17:59 ListAllSecondaryIndexes() for Thread 5: : Index index_company Bucket default
2022/09/03 03:17:59 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
2022/09/03 03:17:59 ListAllSecondaryIndexes() for Thread 2: : Index index_company Bucket default
2022/09/03 03:17:59 ListAllSecondaryIndexes() for Thread 3: : Index index_company Bucket default
2022/09/03 03:17:59 ListAllSecondaryIndexes() for Thread 4: : Index index_company Bucket default
2022-09-03T03:20:00.432+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:32828->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-09-03T03:20:00.619+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:32820->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-09-03T03:20:00.816+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:32816->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-09-03T03:20:00.822+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:32818->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-09-03T03:20:00.970+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:32822->127.0.0.1:9106: use of closed network connection. Kill Pipe.
--- PASS: TestConcurrentScans_SameIndex (199.04s)
2022-09-03T03:20:01.053+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:32814->127.0.0.1:9106: use of closed network connection. Kill Pipe.
=== RUN   TestConcurrentScans_MultipleIndexes
2022/09/03 03:20:01 In TestConcurrentScans_MultipleIndexes()
2022/09/03 03:20:01 Generating JSON docs
2022/09/03 03:20:06 Setting initial JSON docs in KV
2022/09/03 03:21:18 All indexers are active
2022/09/03 03:21:18 Creating multiple indexes
2022/09/03 03:21:18 Index found:  index_company
2022/09/03 03:21:41 Created the secondary index index_age. Waiting for it become active
2022/09/03 03:21:41 Index is 1928971518213654155 now active
2022/09/03 03:22:04 Created the secondary index index_firstname. Waiting for it become active
2022/09/03 03:22:04 Index is 17753041120344197953 now active
2022/09/03 03:22:04 In Range Scan for Thread 3: 
2022/09/03 03:22:04 In Range Scan
2022/09/03 03:22:04 In Range Scan for Thread 1: 
2022/09/03 03:22:04 ListAllSecondaryIndexes() for Thread 3: : Index index_company Bucket default
2022/09/03 03:22:04 ListAllSecondaryIndexes() for Thread 3: : Index index_firstname Bucket default
2022/09/03 03:22:04 ListAllSecondaryIndexes() for Thread 3: : Index index_age Bucket default
2022/09/03 03:22:04 ListAllSecondaryIndexes() for Thread 1: : Index index_age Bucket default
2022/09/03 03:22:04 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
2022/09/03 03:22:04 ListAllSecondaryIndexes() for Thread 1: : Index index_firstname Bucket default
2022-09-03T03:24:05.370+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:34848->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-09-03T03:24:05.739+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:34856->127.0.0.1:9106: use of closed network connection. Kill Pipe.
--- PASS: TestConcurrentScans_MultipleIndexes (245.07s)
=== RUN   TestMutationsWithMultipleIndexBuilds
2022/09/03 03:24:06 In TestMutationsWithMultipleIndexBuilds()
2022/09/03 03:24:06 In DropAllSecondaryIndexes()
2022/09/03 03:24:06 Index found:  index_company
2022-09-03T03:24:06.125+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:34850->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/09/03 03:24:06 Dropped index index_company
2022/09/03 03:24:06 Index found:  index_age
2022/09/03 03:24:06 Dropped index index_age
2022/09/03 03:24:06 Index found:  index_firstname
2022/09/03 03:24:06 Dropped index index_firstname
2022/09/03 03:24:06 Generating JSON docs
2022/09/03 03:24:11 Setting initial JSON docs in KV
2022/09/03 03:25:24 Created the secondary index index_primary. Waiting for it become active
2022/09/03 03:25:24 Index is 5564729906905746070 now active
2022/09/03 03:25:24 Creating multiple indexes in deferred mode
2022/09/03 03:25:24 Build Indexes and wait for indexes to become active: [index_company index_age index_firstname index_lastname]
2022/09/03 03:25:25 Build command issued for the deferred indexes [index_company index_age index_firstname index_lastname]
2022/09/03 03:25:25 Waiting for the index index_company to become active
2022/09/03 03:25:25 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:26 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:27 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:28 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:29 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:30 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:31 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:32 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:33 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:34 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:35 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:36 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:37 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:38 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:39 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:40 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:41 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:42 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:43 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:44 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:45 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:46 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:47 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:48 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:49 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:50 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:51 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:52 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:53 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:54 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:55 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:56 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:57 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:58 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:25:59 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:00 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:01 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:02 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:03 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:04 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:05 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:06 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:07 Waiting for index 7065656284451394897 to go active ...
2022/09/03 03:26:08 Index is 7065656284451394897 now active
2022/09/03 03:26:08 Waiting for the index index_age to become active
2022/09/03 03:26:08 Index is 2652354460186332603 now active
2022/09/03 03:26:08 Waiting for the index index_firstname to become active
2022/09/03 03:26:08 Index is 3100421675208156216 now active
2022/09/03 03:26:08 Waiting for the index index_lastname to become active
2022/09/03 03:26:08 Index is 2198766039175368244 now active
--- PASS: TestMutationsWithMultipleIndexBuilds (122.03s)
PASS
ok  	github.com/couchbase/indexing/secondary/tests/largedatatests	1057.809s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 63135    0 63135    0     0  6218k      0 --:--:-- --:--:-- --:--:-- 6850k

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[ 50%] Built target ns_ui_build_prepare
[ 50%] Building Go Modules target ns_minify_js using Go 1.18.4
[ 50%] Built target ns_minify_js
[ 50%] Building Go Modules target ns_minify_css using Go 1.18.4
[ 50%] Built target ns_minify_css
[ 50%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.4
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.4
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.4
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.4
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.4
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-03 03:26:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:26:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:26:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:26:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-03 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-03 03:27:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1739-rel-EE-enterprise
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-03 03:27:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:18 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-03 03:27:19 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 5510078464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:27:19 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:27:19 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 5337534464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:27:19 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-03 03:27:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:27:20 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 5200060416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:27:20 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:27:21 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '19', 'memoryTotal': 15466930176, 'memoryFree': 5120167936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:27:21 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:27:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:27:22 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-03 03:27:22 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-03 03:27:22 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-03 03:27:22 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:27:27 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-03 03:27:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-03 03:27:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-03 03:27:28 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-03 03:27:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-03 03:27:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-03 03:27:29 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-03 03:27:29 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-03 03:27:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 36.44763879896487, 'mem_free': 13581283328, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 42.37089729309082
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-03 03:27:33 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-03 03:27:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-03 03:27:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:27:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:27:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:27:39 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-03 03:27:39 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-03 03:27:41 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-03 03:27:41 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:27:41 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-03 03:27:41 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-03 03:28:11 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-03 03:28:11 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-03 03:28:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 147.242077ms
2022-09-03 03:28:12 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '61fde68d-ae60-46ec-bfa4-47ef7632a9a5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '147.242077ms', 'executionTime': '147.075129ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 715.729535ms
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-03 03:28:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.23222ms
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.542415ms
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 62.99066ms
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.598990341330073, 'mem_free': 13555765248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-03 03:28:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-03 03:28:19 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-03 03:28:19 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-03 03:28:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-03 03:28:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:28:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-03 03:28:20 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-03 03:28:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 89.584s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1739-rel-EE-enterprise
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.82516033571775, 'mem_free': 13534412800, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-03 03:28:26 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-03 03:28:26 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '92', 'memoryTotal': 15466930176, 'memoryFree': 13534412800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-03 03:28:27 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-03 03:28:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '93', 'memoryTotal': 15466930176, 'memoryFree': 13526605824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:28:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '89', 'memoryTotal': 15466930176, 'memoryFree': 13527351296, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '89', 'memoryTotal': 15466930176, 'memoryFree': 13525794816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:28:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:28:30 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:28:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-03 03:28:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-03 03:28:30 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-03 03:28:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-03 03:28:35 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-03 03:28:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-03 03:28:36 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-03 03:28:36 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-03 03:28:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-03 03:28:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-03 03:28:37 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-03 03:28:37 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-03 03:28:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.50383413209419, 'mem_free': 13510602752, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.61991262435913
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-03 03:28:42 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-03 03:28:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-03 03:28:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:28:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:28:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:28:48 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-03 03:28:48 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-03 03:28:50 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-03 03:28:50 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:28:50 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-03 03:28:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-03 03:29:20 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-03 03:29:20 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-03 03:29:20 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-03 03:29:20 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 67.339472ms
2022-09-03 03:29:20 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '693c5b3d-e68a-4480-9847-93d823ec18f8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.339472ms', 'executionTime': '67.229403ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-03 03:29:20 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-03 03:29:20 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-03 03:29:20 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 710.134931ms
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.820625ms
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-03 03:29:21 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:29:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee19ea8985c3024d85b77cdf033acbfe2ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-03 03:29:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee19ea8985c3024d85b77cdf033acbfe2ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-03 03:29:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 32.140576ms
2022-09-03 03:29:22 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee19ea8985c3024d85b77cdf033acbfe2ajob_title) USING GSI
2022-09-03 03:29:23 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee19ea8985c3024d85b77cdf033acbfe2ajob_title) USING GSI
2022-09-03 03:29:23 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee19ea8985c3024d85b77cdf033acbfe2ajob_title%29+USING+GSI
2022-09-03 03:29:23 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 27.539398ms
2022-09-03 03:29:24 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee19ea8985c3024d85b77cdf033acbfe2ajob_title'
2022-09-03 03:29:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee19ea8985c3024d85b77cdf033acbfe2ajob_title%27
2022-09-03 03:29:24 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.043873ms
2022-09-03 03:29:26 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee19ea8985c3024d85b77cdf033acbfe2ajob_title'
2022-09-03 03:29:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee19ea8985c3024d85b77cdf033acbfe2ajob_title%27
2022-09-03 03:29:26 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.431906ms
2022-09-03 03:29:27 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-03 03:29:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-03 03:29:27 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-03 03:29:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.984361ms
2022-09-03 03:29:27 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '831accca-b0b0-49a5-86c0-a64fe7a8b2e8', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee19ea8985c3024d85b77cdf033acbfe2ajob_title', 'index_id': '7dd2aa95df415bbe', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.984361ms', 'executionTime': '2.862077ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-03 03:29:27 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-03 03:29:27 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-03 03:29:27 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-03 03:29:27 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-03 03:29:27 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-03 03:29:27 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-03 03:29:27 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-03 03:29:27 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-03 03:29:27 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-03 03:29:28 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-03 03:29:28 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-03 03:29:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-03 03:29:28 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 117.15142ms
2022-09-03 03:29:28 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-03 03:29:28 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-03 03:29:29 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-03 03:29:29 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee19ea8985c3024d85b77cdf033acbfe2ajob_title'
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee19ea8985c3024d85b77cdf033acbfe2ajob_title%27
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.866736ms
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee19ea8985c3024d85b77cdf033acbfe2ajob_title ON default USING GSI
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee19ea8985c3024d85b77cdf033acbfe2ajob_title+ON+default+USING+GSI
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 38.928909ms
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee19ea8985c3024d85b77cdf033acbfe2ajob_title'
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee19ea8985c3024d85b77cdf033acbfe2ajob_title%27
2022-09-03 03:29:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.560696ms
2022-09-03 03:29:30 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '5d3abd16-39e5-462e-b33f-37685582170a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.560696ms', 'executionTime': '5.488281ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:29:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.13094ms
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 47.694135ms
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.17003821646371, 'mem_free': 13419683840, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:29:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:29:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:29:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:29:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:29:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:29:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-03 03:29:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-03 03:29:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-03 03:29:36 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-03 03:29:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-03 03:29:36 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-03 03:29:36 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-03 03:29:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-03 03:29:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:29:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-03 03:29:37 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-03 03:29:37 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.023s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_2'}
[2022-09-03 03:29:37,551] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:37,552] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:37,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:37,884] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:29:37,966] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:29:37,967] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-03 03:29:37,968] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:29:38,618] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:29:38,647] - [task:164] INFO -  {'uptime': '164', 'memoryTotal': 15466930176, 'memoryFree': 13419683840, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:29:38,675] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:29:38,675] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:29:38,675] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:29:38,705] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:29:38,743] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:29:38,744] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:29:38,771] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:29:38,772] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:29:38,772] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:29:38,772] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:29:38,821] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:29:38,824] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:38,824] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:39,094] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:39,095] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:29:39,170] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:29:39,171] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:29:39,204] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:29:39,205] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:29:39,333] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:29:39,360] - [task:164] INFO -  {'uptime': '163', 'memoryTotal': 15466930176, 'memoryFree': 13449703424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:29:39,386] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:29:39,414] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:29:39,414] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:29:39,469] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:29:39,473] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:39,473] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:39,754] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:39,755] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:29:39,826] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:29:39,827] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:29:39,858] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:29:39,859] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:29:39,985] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:29:40,012] - [task:164] INFO -  {'uptime': '159', 'memoryTotal': 15466930176, 'memoryFree': 13448593408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:29:40,037] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:29:40,064] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:29:40,065] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:29:40,118] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:29:40,122] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:40,123] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:40,388] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:40,389] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:29:40,456] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:29:40,457] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:29:40,487] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:29:40,488] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:29:40,608] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:29:40,634] - [task:164] INFO -  {'uptime': '159', 'memoryTotal': 15466930176, 'memoryFree': 13448331264, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:29:40,662] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:29:40,692] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:29:40,692] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:29:40,743] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:29:40,746] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:40,747] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:41,037] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:41,038] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:29:41,112] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:29:41,114] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:29:41,144] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:29:41,145] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:29:41,237] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:29:41,669] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:29:46,672] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:29:46,757] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:29:46,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:46,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:47,043] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:47,044] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:29:47,113] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:29:47,114] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:29:47,114] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:29:47,348] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:29:47,403] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:29:47,403] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:29:48,422] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:29:48,755] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:29:49,092] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:29:49,094] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:29:49,150] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:49,150] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:49,411] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:49,415] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:49,416] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:49,662] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:49,667] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:49,667] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:49,918] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:49,923] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:49,923] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:50,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:29:53,947] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:29:53,948] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.57421561900693, 'mem_free': 13435174912, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:29:53,948] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:29:53,949] - [basetestcase:467] INFO - Time to execute basesetup : 16.401273488998413
[2022-09-03 03:29:54,007] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:29:54,007] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:29:54,059] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:29:54,060] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:29:54,112] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:29:54,112] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:29:54,165] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:29:54,166] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:29:54,218] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:29:54,280] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:29:54,280] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:29:54,280] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:29:59,293] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:29:59,296] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:29:59,296] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:29:59,553] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:00,515] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:30:00,689] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:30:03,356] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:30:03,379] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-03 03:30:04,048] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:30:04,048] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:30:04,048] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:30:34,078] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:30:34,108] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:30:34,134] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:30:34,207] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 71.434346ms
[2022-09-03 03:30:34,208] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e47cf4fb-62b6-4f8f-b382-d2d76614b6c7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '71.434346ms', 'executionTime': '71.321834ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:30:34,208] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:30:34,235] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:30:34,261] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:30:34,944] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 680.706031ms
[2022-09-03 03:30:34,944] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:30:35,018] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:30:35,063] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:30:35,072] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.647774ms
[2022-09-03 03:30:35,278] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:30:35,312] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:30:35,336] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:30:35,337] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:30:35,357] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:30:35,417] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:30:36,239] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeface16618f894c6aa0e41a32bf191f55job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:30:36,266] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeface16618f894c6aa0e41a32bf191f55job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:30:36,296] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.255882ms
[2022-09-03 03:30:36,297] - [base_gsi:282] INFO - BUILD INDEX on default(employeeface16618f894c6aa0e41a32bf191f55job_title) USING GSI
[2022-09-03 03:30:37,330] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeface16618f894c6aa0e41a32bf191f55job_title) USING GSI
[2022-09-03 03:30:37,356] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeface16618f894c6aa0e41a32bf191f55job_title%29+USING+GSI
[2022-09-03 03:30:37,385] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.30202ms
[2022-09-03 03:30:38,416] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeface16618f894c6aa0e41a32bf191f55job_title'
[2022-09-03 03:30:38,444] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeface16618f894c6aa0e41a32bf191f55job_title%27
[2022-09-03 03:30:38,454] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.289215ms
[2022-09-03 03:30:39,485] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeface16618f894c6aa0e41a32bf191f55job_title'
[2022-09-03 03:30:39,511] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeface16618f894c6aa0e41a32bf191f55job_title%27
[2022-09-03 03:30:39,520] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.248155ms
[2022-09-03 03:30:39,993] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-03 03:30:40,158] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:30:41,011] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:30:41,523] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:30:41,556] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:30:41,582] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-03 03:30:41,587] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.436923ms
[2022-09-03 03:30:41,587] - [task:3245] INFO - {'requestID': '87683ff7-2d42-4ab7-b593-d5a3a83395af', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeface16618f894c6aa0e41a32bf191f55job_title', 'index_id': '59023244908add2a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.436923ms', 'executionTime': '3.305674ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-03 03:30:41,588] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:30:41,588] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:30:41,588] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:30:41,589] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:30:41,589] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-03 03:30:41,589] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:30:41,590] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:30:41,590] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:30:41,590] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:30:42,590] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:30:42,620] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:30:42,647] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-03 03:30:42,796] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 138.597978ms
[2022-09-03 03:30:42,796] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:30:42,797] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:30:43,612] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:30:43,612] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:30:44,644] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeface16618f894c6aa0e41a32bf191f55job_title'
[2022-09-03 03:30:44,673] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeface16618f894c6aa0e41a32bf191f55job_title%27
[2022-09-03 03:30:44,683] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.686784ms
[2022-09-03 03:30:44,708] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeface16618f894c6aa0e41a32bf191f55job_title ON default USING GSI
[2022-09-03 03:30:44,734] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeface16618f894c6aa0e41a32bf191f55job_title+ON+default+USING+GSI
[2022-09-03 03:30:44,775] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.817489ms
[2022-09-03 03:30:44,810] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeface16618f894c6aa0e41a32bf191f55job_title'
[2022-09-03 03:30:44,836] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeface16618f894c6aa0e41a32bf191f55job_title%27
[2022-09-03 03:30:44,843] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.822502ms
[2022-09-03 03:30:44,843] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2011b1a4-bf85-48a5-aea4-e2ce8ba6034f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.822502ms', 'executionTime': '5.751304ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:30:44,948] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:30:44,951] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:44,951] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:45,303] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:45,358] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:30:45,358] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:30:45,421] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:30:45,477] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:30:45,480] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:45,480] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:45,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:45,886] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:30:45,886] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:30:45,949] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:30:46,005] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:30:46,005] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:30:46,005] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:30:46,030] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:30:46,056] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:30:46,066] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.256991ms
[2022-09-03 03:30:46,093] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:30:46,121] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:30:46,168] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.149522ms
[2022-09-03 03:30:46,241] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:30:46,241] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.6179916225948, 'mem_free': 13322280960, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:30:46,242] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:30:46,247] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:46,247] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:46,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:46,643] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:46,643] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:46,996] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:47,001] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:47,001] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:47,619] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:47,626] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:47,626] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:48,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:52,459] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:30:52,600] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:30:52,940] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:30:52,968] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:30:52,968] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:30:53,028] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:30:53,080] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:30:53,134] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:30:53,135] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:30:53,222] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:30:53,223] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:30:53,249] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:30:53,377] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:30:53,378] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:30:53,403] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:30:53,429] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:30:53,429] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:30:53,456] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:30:53,481] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:30:53,481] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:30:53,507] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:30:53,533] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:30:53,533] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:30:53,563] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:30:53,563] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:30:53,563] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:30:53,564] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 76.069s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_3'}
[2022-09-03 03:30:53,663] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:53,663] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:54,007] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:54,038] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:30:54,117] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:30:54,118] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-03 03:30:54,118] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:30:54,694] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:30:54,721] - [task:164] INFO -  {'uptime': '240', 'memoryTotal': 15466930176, 'memoryFree': 13381165056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:30:54,746] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:30:54,746] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:30:54,747] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:30:54,782] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:30:54,815] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:30:54,816] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:30:54,843] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:30:54,844] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:30:54,844] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:30:54,844] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:30:54,893] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:30:54,896] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:54,896] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:55,208] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:55,209] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:30:55,284] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:30:55,285] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:30:55,315] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:30:55,316] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:30:55,436] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:30:55,463] - [task:164] INFO -  {'uptime': '238', 'memoryTotal': 15466930176, 'memoryFree': 13373153280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:30:55,489] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:30:55,517] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:30:55,517] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:30:55,566] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:30:55,569] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:55,569] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:55,886] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:55,887] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:30:55,961] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:30:55,963] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:30:55,993] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:30:55,994] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:30:56,114] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:30:56,141] - [task:164] INFO -  {'uptime': '235', 'memoryTotal': 15466930176, 'memoryFree': 13307314176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:30:56,166] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:30:56,196] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:30:56,196] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:30:56,249] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:30:56,254] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:56,255] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:56,585] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:56,586] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:30:56,668] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:30:56,670] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:30:56,704] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:30:56,704] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:30:56,825] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:30:56,852] - [task:164] INFO -  {'uptime': '234', 'memoryTotal': 15466930176, 'memoryFree': 13303050240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:30:56,881] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:30:56,910] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:30:56,910] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:30:56,963] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:30:56,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:30:56,967] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:30:57,295] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:30:57,296] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:30:57,374] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:30:57,375] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:30:57,405] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:30:57,406] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:30:57,499] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:30:57,895] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:31:02,899] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:31:02,981] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:31:02,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:31:02,987] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:31:03,326] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:31:03,327] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:31:03,405] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:31:03,406] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:31:03,407] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:31:03,613] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:31:03,665] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:31:03,666] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:31:04,567] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:31:04,892] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:31:05,181] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:31:05,184] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:31:05,245] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:31:05,245] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:31:05,576] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:31:05,581] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:31:05,581] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:31:05,893] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:31:05,898] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:31:05,899] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:31:06,313] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:31:06,322] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:31:06,322] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:31:06,917] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:31:10,940] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:31:10,940] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.6744455338292, 'mem_free': 13386051584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:31:10,940] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:31:10,941] - [basetestcase:467] INFO - Time to execute basesetup : 17.279744148254395
[2022-09-03 03:31:10,995] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:31:10,995] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:31:11,047] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:31:11,048] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:31:11,099] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:31:11,100] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:31:11,153] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:31:11,153] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:31:11,205] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:31:11,264] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:31:11,265] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:31:11,265] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:31:16,277] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:31:16,282] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:31:16,282] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:31:16,598] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:31:17,574] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:31:17,744] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:31:20,342] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:31:20,359] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-03 03:31:21,225] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:31:21,226] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:31:21,226] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:31:51,256] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:31:51,285] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:31:51,311] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:31:51,380] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.724648ms
[2022-09-03 03:31:51,380] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fe5c1751-86bf-4ea6-93b4-48c267249869', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.724648ms', 'executionTime': '66.664353ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:31:51,380] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:31:51,406] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:31:51,431] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:31:52,097] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 663.592346ms
[2022-09-03 03:31:52,097] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:31:52,148] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:31:52,219] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:31:52,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.385923ms
[2022-09-03 03:31:52,431] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:31:52,466] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:31:52,485] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:31:52,485] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:31:52,497] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:31:52,573] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:31:53,394] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee52d5e9e11cf2488783a3ff0f07c3cc13job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:31:53,422] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee52d5e9e11cf2488783a3ff0f07c3cc13job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:31:53,472] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.350574ms
[2022-09-03 03:31:53,473] - [base_gsi:282] INFO - BUILD INDEX on default(employee52d5e9e11cf2488783a3ff0f07c3cc13job_title) USING GSI
[2022-09-03 03:31:54,503] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee52d5e9e11cf2488783a3ff0f07c3cc13job_title) USING GSI
[2022-09-03 03:31:54,529] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee52d5e9e11cf2488783a3ff0f07c3cc13job_title%29+USING+GSI
[2022-09-03 03:31:54,551] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.102553ms
[2022-09-03 03:31:55,583] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee52d5e9e11cf2488783a3ff0f07c3cc13job_title'
[2022-09-03 03:31:55,609] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee52d5e9e11cf2488783a3ff0f07c3cc13job_title%27
[2022-09-03 03:31:55,617] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.043589ms
[2022-09-03 03:31:56,648] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee52d5e9e11cf2488783a3ff0f07c3cc13job_title'
[2022-09-03 03:31:56,674] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee52d5e9e11cf2488783a3ff0f07c3cc13job_title%27
[2022-09-03 03:31:56,684] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.396106ms
[2022-09-03 03:31:57,076] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-03 03:31:57,336] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:31:58,003] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:31:58,687] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:31:58,721] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:31:58,747] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-03 03:31:58,751] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.292472ms
[2022-09-03 03:31:58,751] - [task:3245] INFO - {'requestID': 'e4b05ef8-aac8-4b7c-97e9-5c17ebcbd3d6', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee52d5e9e11cf2488783a3ff0f07c3cc13job_title', 'index_id': 'dfde43cb2508611c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.292472ms', 'executionTime': '2.201679ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-03 03:31:58,752] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:31:58,752] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:31:58,752] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:31:58,753] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:31:58,753] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-03 03:31:58,753] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:31:58,754] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:31:58,754] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:31:58,754] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:31:59,754] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:31:59,785] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:31:59,811] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-03 03:31:59,928] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 107.681715ms
[2022-09-03 03:31:59,929] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:31:59,929] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:32:00,773] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:32:00,774] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:32:01,806] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee52d5e9e11cf2488783a3ff0f07c3cc13job_title'
[2022-09-03 03:32:01,843] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee52d5e9e11cf2488783a3ff0f07c3cc13job_title%27
[2022-09-03 03:32:01,854] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.651906ms
[2022-09-03 03:32:01,883] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee52d5e9e11cf2488783a3ff0f07c3cc13job_title ON default USING GSI
[2022-09-03 03:32:01,911] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee52d5e9e11cf2488783a3ff0f07c3cc13job_title+ON+default+USING+GSI
[2022-09-03 03:32:01,950] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.967ms
[2022-09-03 03:32:01,989] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee52d5e9e11cf2488783a3ff0f07c3cc13job_title'
[2022-09-03 03:32:02,016] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee52d5e9e11cf2488783a3ff0f07c3cc13job_title%27
[2022-09-03 03:32:02,024] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.441402ms
[2022-09-03 03:32:02,024] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '94721fb0-2a4e-4775-ac62-19041a8a81af', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.441402ms', 'executionTime': '6.368527ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:32:02,129] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:32:02,133] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:02,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:02,559] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:02,614] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:32:02,614] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:32:02,689] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:32:02,747] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:32:02,752] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:02,752] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:03,153] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:03,213] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:32:03,214] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:32:03,288] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:32:03,348] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:32:03,348] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:32:03,348] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:32:03,376] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:32:03,403] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:32:03,412] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.584788ms
[2022-09-03 03:32:03,445] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:32:03,474] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:32:03,537] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.717442ms
[2022-09-03 03:32:03,606] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:32:03,607] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.62831714186276, 'mem_free': 13214359552, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:32:03,607] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:32:03,611] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:03,611] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:04,066] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:04,070] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:04,071] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:04,496] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:04,501] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:04,501] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:04,931] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:04,936] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:04,936] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:05,599] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:10,964] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:32:11,111] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:32:12,174] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:32:12,202] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:32:12,202] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:32:12,273] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:32:12,327] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:32:12,384] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:32:12,385] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:32:12,465] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:32:12,466] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:32:12,492] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:32:12,622] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:32:12,622] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:32:12,649] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:32:12,676] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:32:12,676] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:32:12,705] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:32:12,730] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:32:12,731] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:32:12,759] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:32:12,785] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:32:12,786] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:32:12,812] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:32:12,813] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:32:12,813] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:32:12,813] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.205s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_4'}
[2022-09-03 03:32:12,914] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:12,914] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:13,299] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:13,330] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:32:13,409] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:32:13,410] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-03 03:32:13,410] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:32:13,944] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:32:13,973] - [task:164] INFO -  {'uptime': '319', 'memoryTotal': 15466930176, 'memoryFree': 13327183872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:32:13,999] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:32:13,999] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:32:14,000] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:32:14,031] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:32:14,068] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:32:14,069] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:32:14,100] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:32:14,101] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:32:14,101] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:32:14,101] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:32:14,152] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:32:14,154] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:14,155] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:14,547] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:14,548] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:32:14,636] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:32:14,638] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:32:14,669] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:32:14,670] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:32:14,795] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:32:14,822] - [task:164] INFO -  {'uptime': '319', 'memoryTotal': 15466930176, 'memoryFree': 13330145280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:32:14,848] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:32:14,876] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:32:14,876] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:32:14,927] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:32:14,929] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:14,930] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:15,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:15,325] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:32:15,415] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:32:15,416] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:32:15,447] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:32:15,448] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:32:15,567] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:32:15,594] - [task:164] INFO -  {'uptime': '315', 'memoryTotal': 15466930176, 'memoryFree': 13227384832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:32:15,620] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:32:15,648] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:32:15,649] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:32:15,702] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:32:15,706] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:15,706] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:16,083] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:16,084] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:32:16,173] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:32:16,174] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:32:16,205] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:32:16,206] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:32:16,333] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:32:16,362] - [task:164] INFO -  {'uptime': '315', 'memoryTotal': 15466930176, 'memoryFree': 13255524352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:32:16,388] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:32:16,416] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:32:16,417] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:32:16,467] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:32:16,471] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:16,471] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:16,856] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:16,857] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:32:16,955] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:32:16,956] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:32:16,987] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:32:16,988] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:32:17,085] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:32:17,511] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:32:22,517] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:32:22,603] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:32:22,608] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:22,608] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:23,014] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:23,015] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:32:23,102] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:32:23,103] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:32:23,104] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:32:24,202] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:32:24,256] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:32:24,257] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:32:25,149] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:32:25,461] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:32:25,719] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:32:25,722] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:32:25,777] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:25,777] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:26,142] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:26,148] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:26,148] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:26,539] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:26,544] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:26,544] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:27,074] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:27,084] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:27,084] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:27,734] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:32,219] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:32:32,220] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.08907828245432, 'mem_free': 13317881856, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:32:32,220] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:32:32,220] - [basetestcase:467] INFO - Time to execute basesetup : 19.30905795097351
[2022-09-03 03:32:32,272] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:32:32,272] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:32:32,324] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:32:32,324] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:32:32,377] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:32:32,377] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:32:32,431] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:32:32,431] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:32:32,484] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:32:32,543] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:32:32,543] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:32:32,544] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:32:37,556] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:32:37,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:32:37,560] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:32:37,931] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:32:38,909] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:32:39,156] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:32:41,673] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:32:41,688] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-03 03:32:42,377] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:32:42,377] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:32:42,377] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:33:12,398] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:33:12,428] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:33:12,453] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:33:12,523] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.514001ms
[2022-09-03 03:33:12,523] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f1451d6c-0f48-48b0-ac73-0aeee7f8562b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.514001ms', 'executionTime': '67.453278ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:33:12,523] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:33:12,549] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:33:12,574] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:33:13,258] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 681.720333ms
[2022-09-03 03:33:13,258] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:33:13,368] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:33:13,397] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:33:13,405] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.398664ms
[2022-09-03 03:33:13,608] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:33:13,643] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:33:13,657] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:33:13,657] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:33:13,672] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:33:13,741] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:33:14,564] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:33:14,591] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:33:14,634] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.948693ms
[2022-09-03 03:33:14,634] - [base_gsi:282] INFO - BUILD INDEX on default(employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title) USING GSI
[2022-09-03 03:33:15,666] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title) USING GSI
[2022-09-03 03:33:15,693] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title%29+USING+GSI
[2022-09-03 03:33:15,714] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.725068ms
[2022-09-03 03:33:16,745] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title'
[2022-09-03 03:33:16,772] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title%27
[2022-09-03 03:33:16,782] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.449721ms
[2022-09-03 03:33:17,813] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title'
[2022-09-03 03:33:17,840] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title%27
[2022-09-03 03:33:17,849] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.991321ms
[2022-09-03 03:33:18,332] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-03 03:33:18,499] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:33:19,371] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:33:19,555] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-03 03:33:19,556] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:33:19,734] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-03 03:33:19,734] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-03 03:33:19,735] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (3888333512, 0, b'')
[2022-09-03 03:33:19,852] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:33:19,880] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:33:19,906] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-03 03:33:19,910] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.298614ms
[2022-09-03 03:33:19,910] - [task:3245] INFO - {'requestID': '57c9c508-fb8f-4cd0-8b36-6759d9936de7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title', 'index_id': '9258f95aae769228', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.298614ms', 'executionTime': '2.236264ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-03 03:33:19,910] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:33:19,910] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:33:19,911] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:33:19,911] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:33:19,912] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-03 03:33:19,912] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:33:19,912] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:33:19,912] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:33:19,913] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:33:20,912] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:33:20,943] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:33:20,970] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-03 03:33:21,084] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 106.443044ms
[2022-09-03 03:33:21,085] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:33:21,085] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:33:21,895] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:33:21,895] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:33:22,926] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title'
[2022-09-03 03:33:22,953] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title%27
[2022-09-03 03:33:22,963] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.171457ms
[2022-09-03 03:33:22,989] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title ON default USING GSI
[2022-09-03 03:33:23,017] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title+ON+default+USING+GSI
[2022-09-03 03:33:23,065] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.551738ms
[2022-09-03 03:33:23,099] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title'
[2022-09-03 03:33:23,125] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5c769d07e65143a2ac9c5cfc52f73e6cjob_title%27
[2022-09-03 03:33:23,133] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.930347ms
[2022-09-03 03:33:23,133] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8f6b80ac-348c-4cc4-ad26-e0d6f057e52a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.930347ms', 'executionTime': '5.852081ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:33:23,237] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:33:23,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:23,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:23,684] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:23,741] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:33:23,742] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:33:23,826] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:33:23,884] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:33:23,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:23,887] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:24,400] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:24,462] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:33:24,462] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:33:24,544] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:33:24,603] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:33:24,603] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:33:24,604] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:33:24,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:33:24,656] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:33:24,666] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.530941ms
[2022-09-03 03:33:24,695] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:33:24,722] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:33:24,762] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.290801ms
[2022-09-03 03:33:24,837] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:33:24,837] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.85566085257698, 'mem_free': 13179723776, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:33:24,838] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:33:24,842] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:24,842] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:25,346] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:25,351] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:25,351] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:26,103] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:26,110] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:26,111] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:26,895] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:26,903] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:26,903] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:27,706] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:32,243] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:33:32,386] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:33:32,758] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:33:32,786] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:33:32,786] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:33:32,840] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:33:32,894] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:33:32,946] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:33:32,947] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:33:33,030] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:33:33,031] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:33:33,057] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:33:33,185] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:33:33,186] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:33:33,212] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:33:33,238] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:33:33,238] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:33:33,265] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:33:33,291] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:33:33,291] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:33:33,317] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:33:33,343] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:33:33,343] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:33:33,370] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:33:33,370] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:33:33,370] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:33:33,370] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.510s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_5'}
[2022-09-03 03:33:33,471] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:33,471] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:33,904] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:33,934] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:33:34,019] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:33:34,019] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-03 03:33:34,020] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:33:34,498] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:33:34,527] - [task:164] INFO -  {'uptime': '400', 'memoryTotal': 15466930176, 'memoryFree': 13289992192, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:33:34,553] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:33:34,553] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:33:34,554] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:33:34,594] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:33:34,628] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:33:34,629] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:33:34,657] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:33:34,658] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:33:34,658] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:33:34,658] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:33:34,710] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:33:34,713] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:34,714] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:35,176] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:35,177] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:33:35,278] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:33:35,279] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:33:35,312] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:33:35,312] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:33:35,440] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:33:35,467] - [task:164] INFO -  {'uptime': '399', 'memoryTotal': 15466930176, 'memoryFree': 13283770368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:33:35,496] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:33:35,525] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:33:35,525] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:33:35,578] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:33:35,581] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:35,581] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:36,012] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:36,013] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:33:36,107] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:33:36,108] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:33:36,138] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:33:36,139] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:33:36,258] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:33:36,286] - [task:164] INFO -  {'uptime': '395', 'memoryTotal': 15466930176, 'memoryFree': 13202640896, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:33:36,312] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:33:36,340] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:33:36,341] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:33:36,393] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:33:36,397] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:36,397] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:36,811] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:36,812] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:33:36,902] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:33:36,903] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:33:36,933] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:33:36,934] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:33:37,051] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:33:37,078] - [task:164] INFO -  {'uptime': '395', 'memoryTotal': 15466930176, 'memoryFree': 13199745024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:33:37,104] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:33:37,133] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:33:37,133] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:33:37,186] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:33:37,189] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:37,189] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:37,611] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:37,612] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:33:37,702] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:33:37,703] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:33:37,733] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:33:37,734] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:33:37,828] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:33:38,238] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:33:43,243] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:33:43,327] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:33:43,332] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:43,332] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:43,755] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:43,756] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:33:43,860] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:33:43,861] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:33:43,861] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:33:44,944] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:33:44,998] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:33:44,998] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:33:45,904] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:33:46,216] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:33:46,501] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:33:46,504] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:33:46,566] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:46,567] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:46,992] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:46,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:46,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:47,413] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:47,418] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:47,418] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:48,108] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:48,120] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:48,121] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:48,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:33:53,781] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:33:53,781] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.12230271797929, 'mem_free': 13293469696, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:33:53,781] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:33:53,781] - [basetestcase:467] INFO - Time to execute basesetup : 20.31289529800415
[2022-09-03 03:33:53,834] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:33:53,834] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:33:53,893] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:33:53,894] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:33:53,951] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:33:53,951] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:33:54,005] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:33:54,005] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:33:54,056] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:33:54,121] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:33:54,121] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:33:54,122] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:33:59,133] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:33:59,136] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:33:59,136] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:33:59,568] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:34:00,576] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:34:00,834] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:34:03,316] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:34:03,333] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-03 03:34:04,436] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:34:04,436] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:34:04,436] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:34:34,467] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:34:34,497] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:34:34,522] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:34:34,591] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.022896ms
[2022-09-03 03:34:34,591] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '006d7316-fb23-4ed2-8f2d-34a9b34c60f2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.022896ms', 'executionTime': '66.962548ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:34:34,591] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:34:34,616] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:34:34,641] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:34:35,338] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 695.376516ms
[2022-09-03 03:34:35,339] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:34:35,408] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:34:35,446] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:34:35,455] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.747897ms
[2022-09-03 03:34:35,658] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:34:35,701] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:34:35,719] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:34:35,720] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:34:35,735] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:34:35,807] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:34:36,620] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb558a89a3ffb415e825c3e1652b50a49job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:34:36,647] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb558a89a3ffb415e825c3e1652b50a49job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:34:36,692] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.17541ms
[2022-09-03 03:34:36,693] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb558a89a3ffb415e825c3e1652b50a49job_title) USING GSI
[2022-09-03 03:34:37,725] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb558a89a3ffb415e825c3e1652b50a49job_title) USING GSI
[2022-09-03 03:34:37,751] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb558a89a3ffb415e825c3e1652b50a49job_title%29+USING+GSI
[2022-09-03 03:34:37,784] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.099809ms
[2022-09-03 03:34:38,815] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb558a89a3ffb415e825c3e1652b50a49job_title'
[2022-09-03 03:34:38,841] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb558a89a3ffb415e825c3e1652b50a49job_title%27
[2022-09-03 03:34:38,851] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.150012ms
[2022-09-03 03:34:39,883] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb558a89a3ffb415e825c3e1652b50a49job_title'
[2022-09-03 03:34:39,912] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb558a89a3ffb415e825c3e1652b50a49job_title%27
[2022-09-03 03:34:39,925] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.026544ms
[2022-09-03 03:34:40,437] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:34:40,607] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:34:44,116] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:34:44,930] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:34:44,963] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:34:44,989] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-03 03:34:44,994] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.984787ms
[2022-09-03 03:34:44,994] - [task:3245] INFO - {'requestID': 'e541fe28-2eb7-428c-a6ac-514fe5f8c287', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb558a89a3ffb415e825c3e1652b50a49job_title', 'index_id': '15eff1314717862e', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.984787ms', 'executionTime': '2.914124ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-03 03:34:44,995] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:34:44,995] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:34:44,996] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:34:44,996] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:34:44,996] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-03 03:34:44,996] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:34:44,997] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:34:44,997] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:34:44,997] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:34:45,997] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:34:46,027] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:34:46,054] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-03 03:34:46,216] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 147.338961ms
[2022-09-03 03:34:46,216] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:34:46,217] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:34:47,889] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:34:47,890] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:34:48,921] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb558a89a3ffb415e825c3e1652b50a49job_title'
[2022-09-03 03:34:48,948] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb558a89a3ffb415e825c3e1652b50a49job_title%27
[2022-09-03 03:34:48,957] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.612718ms
[2022-09-03 03:34:48,983] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb558a89a3ffb415e825c3e1652b50a49job_title ON default USING GSI
[2022-09-03 03:34:49,009] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeb558a89a3ffb415e825c3e1652b50a49job_title+ON+default+USING+GSI
[2022-09-03 03:34:49,050] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.308665ms
[2022-09-03 03:34:49,086] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb558a89a3ffb415e825c3e1652b50a49job_title'
[2022-09-03 03:34:49,112] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb558a89a3ffb415e825c3e1652b50a49job_title%27
[2022-09-03 03:34:49,120] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.818999ms
[2022-09-03 03:34:49,120] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f9d7deb2-7ffc-4b4f-aec2-f2c36df154ea', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.818999ms', 'executionTime': '5.736498ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:34:49,227] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:34:49,230] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:34:49,230] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:34:49,762] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:34:49,818] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:34:49,818] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:34:49,909] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:34:49,965] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:34:49,968] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:34:49,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:34:50,520] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:34:50,579] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:34:50,579] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:34:50,672] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:34:50,729] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:34:50,729] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:34:50,730] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:34:50,756] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:34:50,782] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:34:50,790] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.659587ms
[2022-09-03 03:34:50,816] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:34:50,843] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:34:50,900] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.918594ms
[2022-09-03 03:34:50,962] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:34:50,962] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.65405053924752, 'mem_free': 13138939904, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:34:50,962] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:34:50,966] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:34:50,966] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:34:51,486] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:34:51,491] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:34:51,491] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:34:52,014] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:34:52,021] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:34:52,021] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:34:52,970] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:34:52,982] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:34:52,982] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:34:53,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:00,119] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:35:00,267] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:35:01,544] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:35:01,573] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:35:01,573] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:35:01,637] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:35:01,698] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:35:01,756] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:35:01,757] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:35:01,845] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:35:01,846] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:35:01,874] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:35:02,021] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:35:02,021] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:35:02,051] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:35:02,079] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:35:02,079] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:35:02,108] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:35:02,136] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:35:02,136] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:35:02,165] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:35:02,193] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:35:02,194] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:35:02,223] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:35:02,223] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:35:02,223] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:35:02,224] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 88.812s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_6'}
[2022-09-03 03:35:02,335] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:02,335] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:02,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:02,943] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:35:03,032] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:35:03,033] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-03 03:35:03,033] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:35:03,361] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:35:03,389] - [task:164] INFO -  {'uptime': '488', 'memoryTotal': 15466930176, 'memoryFree': 13227319296, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:35:03,414] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:35:03,414] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:35:03,415] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:35:03,451] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:35:03,484] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:35:03,484] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:35:03,512] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:35:03,513] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:35:03,513] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:35:03,513] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:35:03,562] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:35:03,564] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:03,565] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:04,087] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:04,088] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:35:04,193] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:35:04,194] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:35:04,226] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:35:04,227] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:35:04,352] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:35:04,379] - [task:164] INFO -  {'uptime': '484', 'memoryTotal': 15466930176, 'memoryFree': 13228298240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:35:04,404] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:35:04,432] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:35:04,432] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:35:04,484] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:35:04,486] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:04,487] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:05,018] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:05,020] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:35:05,129] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:35:05,130] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:35:05,161] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:35:05,162] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:35:05,279] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:35:05,306] - [task:164] INFO -  {'uptime': '486', 'memoryTotal': 15466930176, 'memoryFree': 13228728320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:35:05,332] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:35:05,360] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:35:05,361] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:35:05,412] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:35:05,416] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:05,417] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:05,919] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:05,920] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:35:06,023] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:35:06,024] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:35:06,054] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:35:06,054] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:35:06,170] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:35:06,198] - [task:164] INFO -  {'uptime': '485', 'memoryTotal': 15466930176, 'memoryFree': 13231857664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:35:06,224] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:35:06,251] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:35:06,251] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:35:06,303] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:35:06,306] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:06,306] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:06,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:06,840] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:35:06,940] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:35:06,941] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:35:06,972] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:35:06,972] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:35:07,061] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:35:07,459] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:35:12,464] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:35:12,554] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:35:12,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:12,560] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:13,126] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:13,127] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:35:13,237] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:35:13,238] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:35:13,238] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:35:14,216] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:35:14,273] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-03 03:35:14,274] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:35:15,189] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:35:15,523] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:35:15,823] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:35:15,828] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:35:15,892] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:15,892] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:16,438] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:16,442] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:16,443] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:17,050] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:17,057] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:17,057] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:17,965] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:17,974] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:17,974] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:18,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:24,629] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:35:24,629] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.03277093176494, 'mem_free': 13226893312, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:35:24,630] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:35:24,630] - [basetestcase:467] INFO - Time to execute basesetup : 22.297844648361206
[2022-09-03 03:35:24,692] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:35:24,692] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:35:24,748] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:35:24,748] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:35:24,805] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:35:24,806] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:35:24,863] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:35:24,863] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:35:24,915] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:35:24,975] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:35:24,975] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:35:24,976] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:35:29,983] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:35:29,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:35:29,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:35:30,490] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:35:31,519] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:35:31,792] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:35:34,138] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:35:34,154] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-03 03:35:35,564] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:35:35,564] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:35:35,564] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:36:05,586] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:36:05,617] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:36:05,642] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:36:05,711] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.116411ms
[2022-09-03 03:36:05,711] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b71a0b47-7fa5-44da-a15b-62150ec54aa5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.116411ms', 'executionTime': '66.054195ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:36:05,711] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:36:05,736] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:36:05,761] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:36:06,430] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 667.07916ms
[2022-09-03 03:36:06,431] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:36:06,492] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:36:06,529] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:36:06,539] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.673552ms
[2022-09-03 03:36:06,722] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:36:06,754] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:36:06,769] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:36:06,769] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:36:06,783] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:36:06,876] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:36:07,690] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee65756b1bff634374a424f2c1f656600bjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:36:07,716] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee65756b1bff634374a424f2c1f656600bjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:36:07,748] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.506022ms
[2022-09-03 03:36:07,749] - [base_gsi:282] INFO - BUILD INDEX on default(employee65756b1bff634374a424f2c1f656600bjob_title) USING GSI
[2022-09-03 03:36:08,780] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee65756b1bff634374a424f2c1f656600bjob_title) USING GSI
[2022-09-03 03:36:08,805] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee65756b1bff634374a424f2c1f656600bjob_title%29+USING+GSI
[2022-09-03 03:36:08,832] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.554751ms
[2022-09-03 03:36:09,864] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee65756b1bff634374a424f2c1f656600bjob_title'
[2022-09-03 03:36:09,890] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee65756b1bff634374a424f2c1f656600bjob_title%27
[2022-09-03 03:36:09,900] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.155392ms
[2022-09-03 03:36:10,931] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee65756b1bff634374a424f2c1f656600bjob_title'
[2022-09-03 03:36:10,957] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee65756b1bff634374a424f2c1f656600bjob_title%27
[2022-09-03 03:36:10,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.097659ms
[2022-09-03 03:36:11,482] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-03 03:36:11,657] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:36:12,790] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:36:13,425] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-03 03:36:13,596] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:36:14,908] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:36:15,320] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:36:15,496] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:36:19,120] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:36:19,987] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:36:20,019] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:36:20,044] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-03 03:36:20,049] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.046334ms
[2022-09-03 03:36:20,050] - [task:3245] INFO - {'requestID': '6198473f-680e-4114-bab9-da2a90d45f12', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee65756b1bff634374a424f2c1f656600bjob_title', 'index_id': 'a573708f57192272', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.046334ms', 'executionTime': '2.969356ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-03 03:36:20,050] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:36:20,050] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:36:20,051] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:36:20,051] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:36:20,051] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-03 03:36:20,051] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:36:20,052] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:36:20,052] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:36:20,052] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-03 03:36:21,052] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:36:21,081] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-03 03:36:21,106] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-03 03:36:21,494] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 165.500522ms
[2022-09-03 03:36:21,494] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:36:21,495] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:36:23,146] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:36:23,146] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-03 03:36:24,176] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee65756b1bff634374a424f2c1f656600bjob_title'
[2022-09-03 03:36:24,202] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee65756b1bff634374a424f2c1f656600bjob_title%27
[2022-09-03 03:36:24,211] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.762227ms
[2022-09-03 03:36:24,236] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee65756b1bff634374a424f2c1f656600bjob_title ON default USING GSI
[2022-09-03 03:36:24,261] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee65756b1bff634374a424f2c1f656600bjob_title+ON+default+USING+GSI
[2022-09-03 03:36:24,311] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.699573ms
[2022-09-03 03:36:24,345] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee65756b1bff634374a424f2c1f656600bjob_title'
[2022-09-03 03:36:24,370] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee65756b1bff634374a424f2c1f656600bjob_title%27
[2022-09-03 03:36:24,378] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.552988ms
[2022-09-03 03:36:24,378] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '266c7455-c9ae-483d-8a06-d32e19d93a7a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.552988ms', 'executionTime': '5.495721ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:36:24,479] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:36:24,481] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:24,481] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:25,183] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:25,241] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:36:25,241] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:36:25,354] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:36:25,413] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:36:25,416] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:25,416] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:26,128] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:26,186] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:36:26,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:36:26,333] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:36:26,393] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:36:26,394] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:36:26,394] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:36:26,421] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:36:26,446] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:36:26,456] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.301185ms
[2022-09-03 03:36:26,482] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:36:26,509] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:36:26,558] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.042708ms
[2022-09-03 03:36:26,636] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:36:26,636] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.80610805787097, 'mem_free': 13063155712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:36:26,637] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:36:26,640] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:26,640] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:27,284] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:27,288] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:27,289] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:27,933] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:27,937] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:27,938] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:29,025] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:29,032] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:29,032] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:30,195] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:37,742] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:36:37,885] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:36:38,863] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:36:38,892] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:36:38,892] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:36:38,949] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:36:39,009] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:36:39,073] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:36:39,073] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:36:39,153] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:36:39,153] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:36:39,179] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:36:39,309] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:36:39,310] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:36:39,337] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:36:39,362] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:36:39,362] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:36:39,389] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:36:39,414] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:36:39,415] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:36:39,441] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:36:39,466] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:36:39,466] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:36:39,493] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:36:39,493] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-03 03:36:39,493] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:36:39,494] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 97.213s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_7'}
[2022-09-03 03:36:39,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:39,633] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:40,261] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:40,296] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:36:40,376] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:36:40,376] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-03 03:36:40,377] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:36:40,660] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:36:40,688] - [task:164] INFO -  {'uptime': '586', 'memoryTotal': 15466930176, 'memoryFree': 13167071232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:36:40,717] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:36:40,717] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:36:40,717] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:36:40,751] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:36:40,789] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:36:40,790] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:36:40,818] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:36:40,819] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:36:40,820] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:36:40,820] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:36:40,872] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:36:40,876] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:40,876] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:41,535] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:41,536] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:36:41,653] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:36:41,654] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:36:41,684] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:36:41,685] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:36:41,812] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:36:41,838] - [task:164] INFO -  {'uptime': '585', 'memoryTotal': 15466930176, 'memoryFree': 13167034368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:36:41,864] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:36:41,893] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:36:41,893] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:36:41,943] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:36:41,946] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:41,947] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:42,550] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:42,551] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:36:42,666] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:36:42,667] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:36:42,697] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:36:42,697] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:36:42,817] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:36:42,845] - [task:164] INFO -  {'uptime': '581', 'memoryTotal': 15466930176, 'memoryFree': 13166747648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:36:42,871] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:36:42,899] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:36:42,899] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:36:42,950] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:36:42,955] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:42,955] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:43,568] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:43,569] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:36:43,682] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:36:43,683] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:36:43,713] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:36:43,713] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:36:43,831] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:36:43,860] - [task:164] INFO -  {'uptime': '581', 'memoryTotal': 15466930176, 'memoryFree': 13167013888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:36:43,891] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:36:43,919] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:36:43,920] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:36:43,974] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:36:43,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:43,978] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:44,575] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:44,576] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:36:44,698] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:36:44,700] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:36:44,730] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:36:44,731] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:36:44,819] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:36:45,193] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:36:50,198] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:36:50,284] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:36:50,289] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:50,290] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:50,903] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:50,904] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:36:51,021] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:36:51,022] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:36:51,022] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:36:51,992] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:36:52,047] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:36:52,050] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:36:52,989] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:36:53,312] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:36:53,574] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:36:53,577] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-03 03:36:53,634] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:53,634] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:54,286] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:54,290] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:54,291] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:55,168] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:55,180] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:55,180] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:56,231] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:36:56,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:36:56,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:36:57,309] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:37:04,063] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:37:04,064] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.05034421518767, 'mem_free': 13184811008, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:37:04,064] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:37:04,064] - [basetestcase:467] INFO - Time to execute basesetup : 24.43431782722473
[2022-09-03 03:37:04,114] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:37:04,114] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:37:04,168] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:37:04,168] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:37:04,224] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:37:04,224] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:37:04,277] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:37:04,278] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:37:04,333] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:37:04,395] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:37:04,395] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:37:04,396] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:37:09,408] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:37:09,412] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:37:09,412] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:37:10,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:37:11,146] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:37:11,317] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:37:13,527] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:37:13,608] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:37:13,608] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:37:13,608] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:37:43,637] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:37:43,666] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:37:43,692] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:37:43,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.400264ms
[2022-09-03 03:37:43,759] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '587e7c22-2fe8-4786-bc37-b0d5a5465606', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.400264ms', 'executionTime': '64.335298ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:37:43,759] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:37:43,787] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:37:43,813] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:37:44,502] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 686.954408ms
[2022-09-03 03:37:44,502] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:37:44,574] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:37:44,614] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:37:44,622] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.188962ms
[2022-09-03 03:37:44,831] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:37:44,868] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:37:44,890] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:37:44,891] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:37:44,907] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:37:44,908] - [base_gsi:326] INFO - []
[2022-09-03 03:37:45,794] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:37:45,820] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:37:45,848] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.646041ms
[2022-09-03 03:37:45,885] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:37:45,911] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:37:45,949] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.57078ms
[2022-09-03 03:37:45,950] - [base_gsi:282] INFO - BUILD INDEX on default(employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title,employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr) USING GSI
[2022-09-03 03:37:46,981] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title,employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr) USING GSI
[2022-09-03 03:37:47,007] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title%2Cemployeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr%29+USING+GSI
[2022-09-03 03:37:47,055] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.613634ms
[2022-09-03 03:37:48,087] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title'
[2022-09-03 03:37:48,113] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title%27
[2022-09-03 03:37:48,122] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.91621ms
[2022-09-03 03:37:49,153] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title'
[2022-09-03 03:37:49,179] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title%27
[2022-09-03 03:37:49,188] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.409978ms
[2022-09-03 03:37:49,215] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr'
[2022-09-03 03:37:49,241] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr%27
[2022-09-03 03:37:49,244] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.62538ms
[2022-09-03 03:37:50,278] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title'
[2022-09-03 03:37:50,305] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title%27
[2022-09-03 03:37:50,314] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.720498ms
[2022-09-03 03:37:50,341] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title ON default USING GSI
[2022-09-03 03:37:50,367] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title+ON+default+USING+GSI
[2022-09-03 03:37:50,404] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.599317ms
[2022-09-03 03:37:50,441] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr'
[2022-09-03 03:37:50,467] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr%27
[2022-09-03 03:37:50,475] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.219163ms
[2022-09-03 03:37:50,501] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr ON default USING GSI
[2022-09-03 03:37:50,527] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr+ON+default+USING+GSI
[2022-09-03 03:37:50,573] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.677951ms
[2022-09-03 03:37:50,607] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title'
[2022-09-03 03:37:50,634] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedd0ab686099b4e71a46494c8bf5dc3ecjob_title%27
[2022-09-03 03:37:50,641] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.275063ms
[2022-09-03 03:37:50,641] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e94c0b2c-1c44-4992-babe-224357cd4d94', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.275063ms', 'executionTime': '5.207606ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:37:50,668] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr'
[2022-09-03 03:37:50,694] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedd0ab686099b4e71a46494c8bf5dc3ecjoin_yr%27
[2022-09-03 03:37:50,697] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.305505ms
[2022-09-03 03:37:50,697] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fec7ee2e-b27f-4cde-a732-85d19f22bf83', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.305505ms', 'executionTime': '1.246581ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:37:50,800] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:37:50,803] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:37:50,803] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:37:51,424] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:37:51,481] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:37:51,482] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:37:51,583] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:37:51,642] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:37:51,645] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:37:51,645] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:37:52,256] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:37:52,312] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:37:52,312] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:37:52,415] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:37:52,471] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:37:52,472] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:37:52,472] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:37:52,498] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:37:52,524] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:37:52,533] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.958335ms
[2022-09-03 03:37:52,560] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:37:52,585] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:37:52,631] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.918716ms
[2022-09-03 03:37:52,699] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:37:52,700] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.08352858789563, 'mem_free': 13062119424, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:37:52,700] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:37:52,705] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:37:52,705] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:37:53,340] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:37:53,346] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:37:53,346] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:37:53,990] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:37:53,995] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:37:53,996] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:37:55,000] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:37:55,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:37:55,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:37:56,087] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:03,442] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-03 03:38:03,589] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:38:04,482] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:38:04,510] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:38:04,510] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:38:04,567] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:38:04,620] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:38:04,676] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:38:04,677] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:38:04,769] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:38:04,771] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:38:04,797] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:38:04,927] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:38:04,927] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:38:04,957] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:38:04,983] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:38:04,983] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:38:05,010] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:38:05,036] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:38:05,036] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:38:05,065] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:38:05,091] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:38:05,091] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:38:05,117] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:38:05,117] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-03 03:38:05,117] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:38:05,118] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 85.541s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_8'}
[2022-09-03 03:38:05,199] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:05,199] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:05,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:05,854] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:38:05,932] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:38:05,933] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-03 03:38:05,933] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:38:06,225] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:38:06,252] - [task:164] INFO -  {'uptime': '671', 'memoryTotal': 15466930176, 'memoryFree': 13199253504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:38:06,278] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:38:06,278] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:38:06,279] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:38:06,313] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:38:06,347] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:38:06,348] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:38:06,376] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:38:06,377] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:38:06,377] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:38:06,377] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:38:06,425] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:38:06,428] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:06,428] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:07,032] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:07,033] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:38:07,161] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:38:07,162] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:38:07,192] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:38:07,192] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:38:07,314] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:38:07,341] - [task:164] INFO -  {'uptime': '670', 'memoryTotal': 15466930176, 'memoryFree': 13199929344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:38:07,367] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:38:07,396] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:38:07,396] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:38:07,446] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:38:07,450] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:07,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:08,107] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:08,109] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:38:08,237] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:38:08,238] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:38:08,268] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:38:08,269] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:38:08,390] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:38:08,419] - [task:164] INFO -  {'uptime': '671', 'memoryTotal': 15466930176, 'memoryFree': 13199548416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:38:08,445] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:38:08,473] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:38:08,474] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:38:08,528] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:38:08,533] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:08,533] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:09,148] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:09,149] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:38:09,265] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:38:09,266] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:38:09,299] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:38:09,300] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:38:09,417] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:38:09,443] - [task:164] INFO -  {'uptime': '666', 'memoryTotal': 15466930176, 'memoryFree': 13199929344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:38:09,468] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:38:09,500] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:38:09,500] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:38:09,553] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:38:09,556] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:09,556] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:10,180] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:10,181] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:38:10,307] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:38:10,308] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:38:10,344] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:38:10,345] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:38:10,438] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:38:10,831] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:38:15,837] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:38:15,925] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:38:15,931] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:15,931] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:16,588] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:16,589] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:38:16,711] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:38:16,712] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:38:16,712] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:38:17,601] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:38:17,656] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:38:17,657] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:38:18,757] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:38:19,069] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:38:19,333] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:38:19,336] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-03 03:38:19,390] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:19,390] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:20,009] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:20,016] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:20,016] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:20,894] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:20,903] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:20,903] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:21,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:21,969] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:21,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:23,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:29,834] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:38:29,834] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.34466251211112, 'mem_free': 13192904704, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:38:29,835] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:38:29,835] - [basetestcase:467] INFO - Time to execute basesetup : 24.638603687286377
[2022-09-03 03:38:29,886] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:38:29,886] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:38:29,941] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:38:29,942] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:38:29,998] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:38:29,999] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:38:30,055] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:38:30,055] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:38:30,114] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:38:30,176] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:38:30,176] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:38:30,176] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:38:35,185] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:38:35,188] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:38:35,188] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:38:35,833] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:38:36,884] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:38:37,175] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:38:39,293] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:38:39,372] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:38:39,373] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:38:39,373] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:39:09,384] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:39:09,414] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:39:09,440] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:39:09,507] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.752549ms
[2022-09-03 03:39:09,508] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '522fd815-3fe6-44ec-9aef-3195741e3b12', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.752549ms', 'executionTime': '65.691863ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:39:09,508] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:39:09,534] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:39:09,559] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:39:10,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 659.735328ms
[2022-09-03 03:39:10,221] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:39:10,320] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:39:10,362] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:39:10,371] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.719948ms
[2022-09-03 03:39:10,581] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:39:10,618] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:39:10,639] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:39:10,639] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:39:10,653] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:39:10,654] - [base_gsi:326] INFO - []
[2022-09-03 03:39:11,546] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:39:11,571] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:39:11,605] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.099164ms
[2022-09-03 03:39:11,606] - [base_gsi:282] INFO - BUILD INDEX on default(employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr) USING GSI
[2022-09-03 03:39:12,637] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr) USING GSI
[2022-09-03 03:39:12,663] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr%29+USING+GSI
[2022-09-03 03:39:12,690] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.756361ms
[2022-09-03 03:39:13,721] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr'
[2022-09-03 03:39:13,748] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr%27
[2022-09-03 03:39:13,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.695277ms
[2022-09-03 03:39:14,792] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr'
[2022-09-03 03:39:14,816] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr%27
[2022-09-03 03:39:14,826] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.985365ms
[2022-09-03 03:39:15,857] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr'
[2022-09-03 03:39:15,884] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr%27
[2022-09-03 03:39:15,893] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.918265ms
[2022-09-03 03:39:15,921] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr ON default USING GSI
[2022-09-03 03:39:15,947] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr+ON+default+USING+GSI
[2022-09-03 03:39:15,983] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.477132ms
[2022-09-03 03:39:16,021] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr'
[2022-09-03 03:39:16,050] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f55e369aa1542dd94e0b4c8c707c85cjob_title_join_yr%27
[2022-09-03 03:39:16,057] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.419094ms
[2022-09-03 03:39:16,057] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8af7bdbc-9fbf-4cd8-9ea7-ab9fabee6ecc', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.419094ms', 'executionTime': '5.290083ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:39:16,164] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:39:16,167] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:16,167] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:16,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:16,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:39:16,896] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:39:17,005] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:39:17,064] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:39:17,068] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:17,069] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:17,740] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:17,797] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:39:17,797] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:39:17,913] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:39:17,973] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:39:17,974] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:39:17,974] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:39:18,001] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:39:18,027] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:39:18,036] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.251227ms
[2022-09-03 03:39:18,067] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:39:18,093] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:39:18,142] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.69111ms
[2022-09-03 03:39:18,208] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:39:18,208] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.872912103487773, 'mem_free': 13073940480, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:39:18,209] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:39:18,213] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:18,213] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:18,889] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:18,893] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:18,894] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:19,592] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:19,597] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:19,597] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:20,597] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:20,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:20,605] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:21,773] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:29,347] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-03 03:39:29,501] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:39:30,183] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:39:30,212] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:39:30,212] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:39:30,269] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:39:30,326] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:39:30,387] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:39:30,388] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:39:30,474] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:39:30,475] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:39:30,503] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:39:30,653] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:39:30,653] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:39:30,682] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:39:30,714] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:39:30,714] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:39:30,742] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:39:30,770] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:39:30,770] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:39:30,800] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:39:30,828] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:39:30,828] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:39:30,857] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:39:30,858] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-03 03:39:30,858] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:39:30,858] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 85.720s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_9'}
[2022-09-03 03:39:30,950] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:30,950] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:31,644] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:31,680] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:39:31,763] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:39:31,763] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-03 03:39:31,764] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:39:31,977] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:39:32,006] - [task:164] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 13178179584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:39:32,033] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:39:32,033] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:39:32,033] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:39:32,069] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:39:32,103] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:39:32,104] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:39:32,133] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:39:32,134] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:39:32,134] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:39:32,134] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:39:32,183] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:39:32,186] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:32,186] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:32,845] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:32,846] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:39:32,975] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:39:32,976] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:39:33,009] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:39:33,010] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:39:33,144] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:39:33,170] - [task:164] INFO -  {'uptime': '756', 'memoryTotal': 15466930176, 'memoryFree': 13181779968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:39:33,196] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:39:33,228] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:39:33,229] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:39:33,281] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:39:33,284] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:33,284] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:33,925] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:33,926] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:39:34,053] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:39:34,054] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:39:34,084] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:39:34,085] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:39:34,202] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:39:34,228] - [task:164] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 13180026880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:39:34,253] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:39:34,280] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:39:34,280] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:39:34,331] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:39:34,336] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:34,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:35,025] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:35,026] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:39:35,154] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:39:35,155] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:39:35,186] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:39:35,187] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:39:35,320] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:39:35,348] - [task:164] INFO -  {'uptime': '752', 'memoryTotal': 15466930176, 'memoryFree': 13181829120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:39:35,377] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:39:35,407] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:39:35,408] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:39:35,463] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:39:35,467] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:35,467] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:36,138] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:36,139] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:39:36,263] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:39:36,264] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:39:36,294] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:39:36,295] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:39:36,386] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:39:36,784] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:39:41,789] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:39:41,872] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:39:41,876] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:41,876] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:42,528] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:42,529] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:39:42,660] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:39:42,661] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:39:42,661] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:39:43,522] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:39:43,576] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:39:43,577] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:39:44,487] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:39:44,799] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:39:45,058] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:39:45,061] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-03 03:39:45,116] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:45,116] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:45,746] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:45,751] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:45,751] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:46,714] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:46,726] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:46,727] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:47,832] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:47,845] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:39:47,845] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:39:48,964] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:39:55,743] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:39:55,743] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.06352682311086, 'mem_free': 13176598528, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:39:55,743] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:39:55,744] - [basetestcase:467] INFO - Time to execute basesetup : 24.796491384506226
[2022-09-03 03:39:55,797] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:39:55,798] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:39:55,856] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:39:55,857] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:39:55,912] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:39:55,912] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:39:55,965] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:39:55,965] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:39:56,016] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:39:56,076] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:39:56,077] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:39:56,077] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:40:01,086] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:40:01,089] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:01,090] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:01,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:02,906] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:40:03,081] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:40:05,194] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:40:05,271] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:40:05,271] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:40:05,271] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:40:35,307] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:40:35,336] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:40:35,363] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:40:35,431] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.074893ms
[2022-09-03 03:40:35,431] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5209e7be-b52f-4e2c-9d8a-af2ea25891eb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.074893ms', 'executionTime': '66.012629ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:40:35,431] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:40:35,458] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:40:35,484] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:40:36,158] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 671.717893ms
[2022-09-03 03:40:36,158] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:40:36,247] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:40:36,282] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:40:36,290] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.362119ms
[2022-09-03 03:40:36,507] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:40:36,545] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:40:36,562] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:40:36,562] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:40:36,577] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:40:36,651] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:40:37,462] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5382fbcb57454bcdb77b7b05dd736630join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:40:37,490] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5382fbcb57454bcdb77b7b05dd736630join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:40:37,527] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.795975ms
[2022-09-03 03:40:37,528] - [base_gsi:282] INFO - BUILD INDEX on default(`employee5382fbcb57454bcdb77b7b05dd736630join_yr`) USING GSI
[2022-09-03 03:40:38,561] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee5382fbcb57454bcdb77b7b05dd736630join_yr`) USING GSI
[2022-09-03 03:40:38,588] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee5382fbcb57454bcdb77b7b05dd736630join_yr%60%29+USING+GSI
[2022-09-03 03:40:38,612] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.641491ms
[2022-09-03 03:40:38,655] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5382fbcb57454bcdb77b7b05dd736630join_yr'
[2022-09-03 03:40:38,686] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5382fbcb57454bcdb77b7b05dd736630join_yr%27
[2022-09-03 03:40:38,695] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.394758ms
[2022-09-03 03:40:39,728] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5382fbcb57454bcdb77b7b05dd736630join_yr'
[2022-09-03 03:40:39,753] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5382fbcb57454bcdb77b7b05dd736630join_yr%27
[2022-09-03 03:40:39,764] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.924997ms
[2022-09-03 03:40:40,794] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5382fbcb57454bcdb77b7b05dd736630join_yr'
[2022-09-03 03:40:40,819] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5382fbcb57454bcdb77b7b05dd736630join_yr%27
[2022-09-03 03:40:40,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.896036ms
[2022-09-03 03:40:41,859] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:40:41,885] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-03 03:40:41,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.49721ms
[2022-09-03 03:40:41,890] - [base_gsi:504] INFO - {'requestID': 'b4cdafac-d3e9-49af-8683-ae4d7d5bf572', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5382fbcb57454bcdb77b7b05dd736630join_yr', 'index_id': '4627ed1cc33a285a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.49721ms', 'executionTime': '3.436247ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-09-03 03:40:41,891] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:40:41,891] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:40:41,891] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:40:41,891] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:40:41,892] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:40:41,893] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:40:41,893] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:40:41,893] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:40:41,957] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:40:41,997] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:40:42,022] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-03 03:40:42,251] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 215.883736ms
[2022-09-03 03:40:42,252] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:40:42,252] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:40:44,080] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:40:44,106] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5382fbcb57454bcdb77b7b05dd736630join_yr'
[2022-09-03 03:40:44,132] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5382fbcb57454bcdb77b7b05dd736630join_yr%27
[2022-09-03 03:40:44,137] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.353191ms
[2022-09-03 03:40:44,138] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2c4d9a86-2684-4635-a89b-f65c7555c903', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.353191ms', 'executionTime': '4.288341ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:40:44,238] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:40:44,242] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:44,242] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:44,969] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:45,025] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:40:45,026] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:40:45,134] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:40:45,192] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:40:45,195] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:45,195] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:45,875] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:45,931] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:40:45,932] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:40:46,044] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:40:46,104] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:40:46,104] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:40:46,104] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:40:46,129] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:40:46,155] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:40:46,160] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.588566ms
[2022-09-03 03:40:46,160] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '45fc515b-faa5-463f-91fc-8ee21480e163', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.588566ms', 'executionTime': '3.529778ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:40:46,214] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:40:46,214] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.14212947351284, 'mem_free': 13035937792, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:40:46,214] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:40:46,218] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:46,218] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:46,888] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:46,892] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:46,893] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:47,546] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:47,551] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:47,551] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:48,446] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:48,458] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:48,458] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:49,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:57,588] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-03 03:40:57,723] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:40:57,776] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:40:57,831] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:40:57,884] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:40:57,884] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:40:57,959] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:40:57,960] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:40:57,986] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:40:58,115] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:40:58,115] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:40:58,143] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:40:58,169] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:40:58,169] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:40:58,195] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:40:58,221] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:40:58,221] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:40:58,247] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:40:58,273] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:40:58,273] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:40:58,300] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:40:58,300] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-03 03:40:58,300] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:40:58,301] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 87.406s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_10'}
[2022-09-03 03:40:58,381] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:58,381] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:40:59,053] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:40:59,094] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:40:59,182] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:40:59,183] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-03 03:40:59,184] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:40:59,407] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:40:59,435] - [task:164] INFO -  {'uptime': '846', 'memoryTotal': 15466930176, 'memoryFree': 13187104768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:40:59,461] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:40:59,461] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:40:59,461] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:40:59,502] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:40:59,534] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:40:59,534] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:40:59,563] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:40:59,564] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:40:59,564] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:40:59,564] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:40:59,613] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:40:59,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:40:59,616] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:00,326] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:00,327] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:41:00,460] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:41:00,461] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:41:00,492] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:41:00,493] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:41:00,622] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:41:00,650] - [task:164] INFO -  {'uptime': '841', 'memoryTotal': 15466930176, 'memoryFree': 13186134016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:41:00,677] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:41:00,706] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:41:00,706] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:41:00,760] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:41:00,765] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:00,766] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:01,468] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:01,469] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:41:01,603] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:41:01,604] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:41:01,635] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:41:01,636] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:41:01,753] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:41:01,780] - [task:164] INFO -  {'uptime': '842', 'memoryTotal': 15466930176, 'memoryFree': 13186068480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:41:01,806] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:41:01,834] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:41:01,834] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:41:01,886] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:41:01,889] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:01,889] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:02,612] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:02,613] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:41:02,750] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:41:02,751] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:41:02,786] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:41:02,787] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:41:02,911] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:41:02,938] - [task:164] INFO -  {'uptime': '842', 'memoryTotal': 15466930176, 'memoryFree': 13185589248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:41:02,964] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:41:02,992] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:41:02,993] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:41:03,047] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:41:03,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:03,051] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:03,708] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:03,709] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:41:03,836] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:41:03,838] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:41:03,869] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:41:03,870] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:41:03,962] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:41:04,349] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:41:09,351] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:41:09,436] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:41:09,452] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:09,452] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:10,145] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:10,146] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:41:10,280] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:41:10,281] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:41:10,282] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:41:11,137] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:41:11,191] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:41:11,192] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:41:12,097] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:41:12,418] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:41:12,679] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:41:12,683] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-03 03:41:12,737] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:12,737] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:13,402] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:13,407] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:13,407] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:14,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:14,438] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:14,438] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:15,586] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:15,594] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:15,594] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:16,752] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:23,843] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:41:23,844] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 28.03215834922467, 'mem_free': 13180743680, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:41:23,844] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:41:23,844] - [basetestcase:467] INFO - Time to execute basesetup : 25.46587109565735
[2022-09-03 03:41:23,900] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:41:23,901] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:41:23,963] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:41:23,963] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:41:24,022] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:41:24,022] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:41:24,080] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:41:24,081] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:41:24,134] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:41:24,198] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:41:24,198] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:41:24,198] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:41:29,211] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:41:29,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:41:29,216] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:41:29,880] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:41:30,939] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:41:31,234] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:41:34,346] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:41:34,424] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:41:34,425] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:41:34,425] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:42:04,449] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:42:04,479] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:42:04,504] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:42:04,573] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.518372ms
[2022-09-03 03:42:04,573] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '85b06c24-3945-467a-abf6-db65af0c5462', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.518372ms', 'executionTime': '67.45002ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:42:04,574] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:42:04,600] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:42:04,625] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:42:05,320] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 693.681026ms
[2022-09-03 03:42:05,321] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:42:05,409] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:42:05,451] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:42:05,459] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.390905ms
[2022-09-03 03:42:05,668] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:42:05,706] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:42:05,725] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:42:05,725] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:42:05,737] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:42:05,803] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:42:06,624] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:42:06,650] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:42:06,690] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.380881ms
[2022-09-03 03:42:06,690] - [base_gsi:282] INFO - BUILD INDEX on default(`employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr`) USING GSI
[2022-09-03 03:42:07,721] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr`) USING GSI
[2022-09-03 03:42:07,747] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr%60%29+USING+GSI
[2022-09-03 03:42:07,769] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.406578ms
[2022-09-03 03:42:07,813] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr'
[2022-09-03 03:42:07,844] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr%27
[2022-09-03 03:42:07,853] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.149255ms
[2022-09-03 03:42:08,884] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr'
[2022-09-03 03:42:08,909] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr%27
[2022-09-03 03:42:08,919] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.920313ms
[2022-09-03 03:42:09,950] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr'
[2022-09-03 03:42:09,975] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr%27
[2022-09-03 03:42:09,985] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.535844ms
[2022-09-03 03:42:11,016] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:42:11,043] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-03 03:42:11,048] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.487306ms
[2022-09-03 03:42:11,048] - [base_gsi:504] INFO - {'requestID': 'ac5da869-d600-46c4-8213-48e1d2166f20', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr', 'index_id': '5f1ee3225314bd94', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.487306ms', 'executionTime': '3.404004ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-09-03 03:42:11,048] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:42:11,049] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:42:11,049] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:42:11,049] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:42:11,049] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:42:11,050] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:42:11,050] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:42:11,050] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:42:11,125] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:42:11,165] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:42:11,190] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-03 03:42:11,359] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 156.497325ms
[2022-09-03 03:42:11,359] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:42:11,360] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:42:12,630] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-03 03:42:12,682] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-03 03:42:12,713] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:42:12,740] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-03 03:42:12,745] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.811461ms
[2022-09-03 03:42:12,746] - [base_gsi:504] INFO - {'requestID': 'edd52adb-bbbf-4699-838d-ce70883dd0e7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr', 'index_id': '5f1ee3225314bd94', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.811461ms', 'executionTime': '1.74523ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-09-03 03:42:12,746] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:42:12,746] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:42:12,747] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:42:12,747] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:42:12,747] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:42:12,748] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:42:12,748] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:42:12,748] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:42:12,809] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:42:12,850] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:42:12,875] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-03 03:42:12,996] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 109.713689ms
[2022-09-03 03:42:12,996] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:42:12,997] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:42:14,267] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr ON default USING GSI
[2022-09-03 03:42:14,293] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr+ON+default+USING+GSI
[2022-09-03 03:42:14,333] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.609641ms
[2022-09-03 03:42:14,369] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr'
[2022-09-03 03:42:14,395] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3afe72d24b9b46dfbc6faa6b289b9d08join_yr%27
[2022-09-03 03:42:14,402] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.299553ms
[2022-09-03 03:42:14,402] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'dabc7b21-7b28-491a-b8c5-7607a3e2a5b3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.299553ms', 'executionTime': '5.240155ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:42:14,509] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:42:14,512] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:14,512] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:15,247] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:15,316] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:42:15,316] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:42:15,440] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:42:15,505] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:42:15,508] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:15,508] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:16,256] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:16,313] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:42:16,313] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:42:16,436] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:42:16,499] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:42:16,499] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:42:16,499] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:42:16,524] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:42:16,549] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:42:16,561] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.31358ms
[2022-09-03 03:42:16,586] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:42:16,611] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:42:16,660] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.206304ms
[2022-09-03 03:42:16,729] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:42:16,729] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.36666461342901, 'mem_free': 13015134208, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:42:16,729] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:42:16,733] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:16,733] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:17,452] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:17,457] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:17,457] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:18,344] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:18,352] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:18,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:19,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:19,541] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:19,541] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:20,816] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:28,331] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-03 03:42:28,472] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:42:29,702] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:42:29,730] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:42:29,731] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:42:29,791] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:42:29,844] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:42:29,899] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:42:29,900] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:42:29,978] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:42:29,979] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:42:30,005] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:42:30,138] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:42:30,138] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:42:30,166] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:42:30,191] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:42:30,191] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:42:30,218] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:42:30,243] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:42:30,243] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:42:30,270] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:42:30,296] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:42:30,296] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:42:30,325] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:42:30,325] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-03 03:42:30,325] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:42:30,326] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 92.001s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_11'}
[2022-09-03 03:42:30,407] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:30,407] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:31,091] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:31,125] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:42:31,208] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:42:31,208] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-03 03:42:31,209] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:42:31,433] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:42:31,460] - [task:164] INFO -  {'uptime': '937', 'memoryTotal': 15466930176, 'memoryFree': 13157965824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:42:31,486] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:42:31,486] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:42:31,486] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:42:31,521] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:42:31,568] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:42:31,569] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:42:31,596] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:42:31,597] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:42:31,597] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:42:31,597] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:42:31,647] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:42:31,650] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:31,650] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:32,337] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:32,338] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:42:32,466] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:42:32,467] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:42:32,497] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:42:32,497] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:42:32,620] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:42:32,647] - [task:164] INFO -  {'uptime': '936', 'memoryTotal': 15466930176, 'memoryFree': 13160329216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:42:32,673] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:42:32,702] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:42:32,702] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:42:32,755] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:42:32,758] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:32,758] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:33,446] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:33,447] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:42:33,580] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:42:33,581] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:42:33,614] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:42:33,615] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:42:33,741] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:42:33,769] - [task:164] INFO -  {'uptime': '933', 'memoryTotal': 15466930176, 'memoryFree': 13159653376, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:42:33,796] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:42:33,824] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:42:33,825] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:42:33,879] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:42:33,882] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:33,882] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:34,625] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:34,627] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:42:34,780] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:42:34,781] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:42:34,812] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:42:34,813] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:42:34,937] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:42:34,965] - [task:164] INFO -  {'uptime': '932', 'memoryTotal': 15466930176, 'memoryFree': 13160980480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:42:34,993] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:42:35,022] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:42:35,022] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:42:35,075] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:42:35,078] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:35,078] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:35,808] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:35,809] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:42:35,940] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:42:35,941] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:42:35,972] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:42:35,972] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:42:36,065] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:42:36,503] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:42:41,508] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:42:41,597] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:42:41,602] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:41,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:42,355] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:42,356] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:42:42,485] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:42:42,486] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:42:42,487] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:42:43,238] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:42:43,293] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-03 03:42:43,293] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:42:44,199] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:42:44,512] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:42:44,785] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:42:44,788] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-03 03:42:44,842] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:44,842] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:45,564] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:45,569] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:45,569] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:46,671] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:46,680] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:46,680] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:47,924] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:47,932] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:42:47,932] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:42:49,094] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:42:56,436] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:42:56,437] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.18205809494265, 'mem_free': 13157695488, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:42:56,437] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:42:56,437] - [basetestcase:467] INFO - Time to execute basesetup : 26.03271484375
[2022-09-03 03:42:56,488] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:42:56,488] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:42:56,541] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:42:56,541] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:42:56,595] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:42:56,595] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:42:56,652] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:42:56,652] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:42:56,706] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:42:56,767] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:42:56,768] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:42:56,768] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:43:01,780] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:43:01,783] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:43:01,783] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:43:02,512] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:43:03,727] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:43:03,896] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:43:06,862] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:43:06,942] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:06,943] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:43:06,943] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:43:36,966] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:43:36,996] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:43:37,023] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:43:37,094] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.158996ms
[2022-09-03 03:43:37,094] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e50b4131-2613-4be9-a959-fe51172503a5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.158996ms', 'executionTime': '69.098166ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:43:37,095] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:43:37,121] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:43:37,148] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:43:37,846] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 695.274876ms
[2022-09-03 03:43:37,846] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:43:37,908] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:43:37,950] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:43:37,958] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.171302ms
[2022-09-03 03:43:38,152] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:38,190] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:43:38,222] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:43:38,223] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:43:38,247] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:43:38,311] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:39,113] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-03 03:43:39,138] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-03 03:43:39,180] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.432767ms
[2022-09-03 03:43:39,180] - [base_gsi:282] INFO - BUILD INDEX on default(`employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr`) USING GSI
[2022-09-03 03:43:40,214] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr`) USING GSI
[2022-09-03 03:43:40,241] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr%60%29+USING+GSI
[2022-09-03 03:43:40,263] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.494574ms
[2022-09-03 03:43:40,320] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr'
[2022-09-03 03:43:40,357] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr%27
[2022-09-03 03:43:40,366] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.139361ms
[2022-09-03 03:43:41,397] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr'
[2022-09-03 03:43:41,423] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr%27
[2022-09-03 03:43:41,433] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.321165ms
[2022-09-03 03:43:42,465] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr'
[2022-09-03 03:43:42,492] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr%27
[2022-09-03 03:43:42,502] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.872292ms
[2022-09-03 03:43:43,533] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:43:43,560] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-03 03:43:43,564] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.819528ms
[2022-09-03 03:43:43,565] - [base_gsi:504] INFO - {'requestID': 'f9a547cf-db17-46ee-a22d-4dd6cedcde96', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr', 'index_id': '313598ff1e9bc255', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.819528ms', 'executionTime': '2.745451ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-09-03 03:43:43,565] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-03 03:43:43,566] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:43:43,566] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-03 03:43:43,566] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-03 03:43:43,566] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:43:43,566] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:43:43,567] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:43:43,567] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-03 03:43:43,631] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-03 03:43:43,672] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-03 03:43:43,698] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-03 03:43:43,843] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 130.235129ms
[2022-09-03 03:43:43,843] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-03 03:43:43,844] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-03 03:43:45,790] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-03 03:43:48,602] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:43:48,659] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-03 03:43:48,660] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:43:49,579] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:43:49,907] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:43:50,199] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:43:50,205] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-03 03:43:52,238] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:43:52,238] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:43:52,266] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:43:52,296] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:43:52,296] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:43:52,324] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:43:52,353] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:43:52,354] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:43:52,381] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:43:52,410] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:43:52,410] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:43:52,438] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:43:52,491] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:52,516] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr'
[2022-09-03 03:43:52,543] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr%27
[2022-09-03 03:43:52,588] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.8816ms
[2022-09-03 03:43:52,589] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0764998e-b9c0-49b9-a5dd-aa733a90ae6e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '43.8816ms', 'executionTime': '43.803633ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:43:52,616] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr'
[2022-09-03 03:43:52,642] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee477fe214b50d4ee0ad49b8ceaf4ba24cjoin_yr%27
[2022-09-03 03:43:52,645] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.325991ms
[2022-09-03 03:43:52,645] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '84b076b5-b4ea-4331-b811-d7f221217204', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.325991ms', 'executionTime': '1.249287ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:43:52,700] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:52,836] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:52,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:43:52,839] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:43:53,607] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:43:53,669] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:43:53,670] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:43:53,793] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:43:53,850] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:53,853] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:43:53,853] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:43:54,634] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:43:54,696] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:43:54,696] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:43:54,823] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:43:54,883] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:43:54,883] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:43:54,884] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:43:54,910] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:43:54,936] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:43:54,947] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.402987ms
[2022-09-03 03:43:54,947] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '564d236b-5576-4102-b37d-1ec0217c2f04', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.402987ms', 'executionTime': '8.320546ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:43:54,947] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:43:54,974] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:43:55,000] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:43:55,004] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.839717ms
[2022-09-03 03:43:55,004] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5f0a65f8-84da-4677-a562-151a280b5e41', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.839717ms', 'executionTime': '1.781519ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:43:55,057] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:43:55,057] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.42798091987466, 'mem_free': 12997287936, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:43:55,057] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:43:55,061] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:43:55,061] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:43:55,834] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:43:55,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:43:55,839] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:43:56,941] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:43:56,954] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:43:56,955] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:43:58,280] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:43:58,288] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:43:58,288] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:43:59,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:07,517] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-03 03:44:07,657] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:44:07,981] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:44:08,009] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:44:08,010] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:44:08,065] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:44:08,118] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:44:08,171] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:44:08,172] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:44:08,251] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:44:08,252] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:44:08,278] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:44:08,409] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:44:08,409] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:44:08,436] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:44:08,462] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:44:08,462] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:44:08,490] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:44:08,516] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:44:08,516] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:44:08,544] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:44:08,573] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:44:08,573] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:44:08,601] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:44:08,601] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-03 03:44:08,601] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:44:08,602] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 98.249s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-03_03-26-50/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-03 03:44:08,660] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:08,660] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:09,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:09,490] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:44:09,578] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1739-rel-EE-enterprise
[2022-09-03 03:44:09,578] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-03 03:44:09,579] - [basetestcase:224] INFO - initializing cluster
[2022-09-03 03:44:09,689] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-03 03:44:09,721] - [task:164] INFO -  {'uptime': '1035', 'memoryTotal': 15466930176, 'memoryFree': 13146619904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:44:09,749] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-03 03:44:09,750] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-03 03:44:09,750] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-03 03:44:09,781] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-03 03:44:09,822] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-03 03:44:09,823] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-03 03:44:09,853] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-03 03:44:09,854] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-03 03:44:09,854] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-03 03:44:09,855] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-03 03:44:09,905] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:44:09,907] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:09,907] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:10,678] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:10,679] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:44:10,823] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:44:10,824] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:44:10,856] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:44:10,857] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:44:10,992] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-03 03:44:11,022] - [task:164] INFO -  {'uptime': '1032', 'memoryTotal': 15466930176, 'memoryFree': 13146701824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:44:11,049] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:44:11,083] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-03 03:44:11,083] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-03 03:44:11,138] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:44:11,141] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:11,141] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:11,908] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:11,910] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:44:12,060] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:44:12,061] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:44:12,093] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:44:12,094] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:44:12,217] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-03 03:44:12,245] - [task:164] INFO -  {'uptime': '1033', 'memoryTotal': 15466930176, 'memoryFree': 13146669056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:44:12,273] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:44:12,304] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-03 03:44:12,304] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-03 03:44:12,358] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:44:12,362] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:12,362] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:13,137] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:13,138] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:44:13,276] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:44:13,277] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:44:13,307] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:44:13,307] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:44:13,426] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-03 03:44:13,452] - [task:164] INFO -  {'uptime': '1033', 'memoryTotal': 15466930176, 'memoryFree': 13147107328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1739-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-03 03:44:13,478] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-03 03:44:13,505] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-03 03:44:13,506] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-03 03:44:13,557] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-03 03:44:13,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:13,560] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:14,297] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:14,298] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:44:14,430] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:44:14,432] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:44:14,461] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-03 03:44:14,462] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-03 03:44:14,552] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-03 03:44:14,935] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:44:19,940] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-03 03:44:20,028] - [basetestcase:262] INFO - done initializing cluster
[2022-09-03 03:44:20,034] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:20,034] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:20,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:20,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-03 03:44:20,906] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-03 03:44:20,907] - [remote_util:5249] INFO - b'ok'
[2022-09-03 03:44:20,908] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-03 03:44:21,672] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-03 03:44:21,730] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-03 03:44:21,731] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-03 03:44:22,669] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:44:23,016] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:44:23,286] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-03 03:44:23,289] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-03 03:44:23,344] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:23,344] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:24,112] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:24,116] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:24,117] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:25,309] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:25,321] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:25,321] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:26,604] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:26,612] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:26,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:27,912] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:35,332] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:44:35,333] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.30646662755299, 'mem_free': 13144772608, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:44:35,333] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:44:35,333] - [basetestcase:467] INFO - Time to execute basesetup : 26.67579412460327
[2022-09-03 03:44:35,384] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:44:35,385] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:44:35,438] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:44:35,438] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:44:35,492] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:44:35,492] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:44:35,544] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-03 03:44:35,544] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-03 03:44:35,596] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:44:35,653] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-03 03:44:35,654] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-03 03:44:35,654] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-03 03:44:40,666] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-03 03:44:40,670] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:44:40,670] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:44:41,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:44:42,633] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-03 03:44:42,824] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-03 03:44:45,744] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-03 03:44:45,824] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:44:45,824] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-03 03:44:45,824] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-03 03:45:15,848] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-03 03:45:15,878] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:45:15,904] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:45:15,971] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.57549ms
[2022-09-03 03:45:15,972] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '35e9c326-781a-48fc-9145-ffdf0210667e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.57549ms', 'executionTime': '65.526666ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-03 03:45:15,972] - [tuq_helper:758] INFO - Create primary index
[2022-09-03 03:45:15,998] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-03 03:45:16,023] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-03 03:45:16,671] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 645.973126ms
[2022-09-03 03:45:16,672] - [tuq_helper:760] INFO - Check if index is online
[2022-09-03 03:45:16,726] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:45:16,767] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:45:16,777] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.410857ms
[2022-09-03 03:45:16,969] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:45:17,003] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-03 03:45:17,031] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-03 03:45:17,033] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-03 03:45:17,052] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-03 03:45:17,143] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:45:17,297] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:45:17,300] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:45:17,300] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:45:18,146] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:45:18,202] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:45:18,203] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-03 03:45:18,332] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:45:18,389] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:45:18,392] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:45:18,392] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:45:19,174] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:45:19,229] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-03 03:45:19,229] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-03 03:45:19,360] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-03 03:45:19,419] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-03 03:45:19,419] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-03 03:45:19,419] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:45:19,445] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-03 03:45:19,471] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-03 03:45:19,480] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.759291ms
[2022-09-03 03:45:19,506] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-03 03:45:19,532] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-03 03:45:19,596] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.442848ms
[2022-09-03 03:45:19,656] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-03 03:45:19,656] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.60492895237043, 'mem_free': 12952555520, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-03 03:45:19,656] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-03 03:45:19,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:45:19,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:45:20,518] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:45:20,523] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:45:20,523] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:45:21,326] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:45:21,331] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:45:21,331] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:45:22,582] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:45:22,590] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-03 03:45:22,590] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-03 03:45:24,005] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-03 03:45:32,919] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-03 03:45:33,078] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-03 03:45:34,327] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-03 03:45:34,355] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-03 03:45:34,355] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-03 03:45:34,414] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:45:34,466] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:45:34,520] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-03 03:45:34,521] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-03 03:45:34,601] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-03 03:45:34,602] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-03 03:45:34,626] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:45:34,754] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-03 03:45:34,755] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:45:34,782] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-03 03:45:34,808] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-03 03:45:34,808] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:45:34,834] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-03 03:45:34,860] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-03 03:45:34,860] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:45:34,886] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-03 03:45:34,911] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-03 03:45:34,911] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-03 03:45:34,937] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-03 03:45:34,938] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-03 03:45:34,938] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-03 03:45:34,939] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 86.358s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91:  8548 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Failed: Required test failed

Binary file /opt/build/test.log matches
Binary file /opt/build/test.log matches
Version: versions-03.09.2022-01.31.cfg
Build Log: make-03.09.2022-01.31.log
Server Log: logs-03.09.2022-01.31.tar.gz

Finished